I've been building webtomarkdown.com to have a tool that allows me to convert various type of content to Markdown so I can pass it to LLMs (Markdown is the language of LLMs).

I started with microsoft/markitdown which is a really nice and fast library. But it does poorly on PDF that are beyond basic text e.g. tables, images etc. In my research and trying it out I see the DS4SD/docling does a lot better in extracting content from PDF. One of the reason it is able to do it is because they trained a neural network based model huggingface/ds4sd/docling-models which takes an image and extract the content.

Running it requires GPU and Cloud-based GPUs, come with a hefty price tag. I don't want to pay for it, when the primary user of the tool is myself and maybe few who explore it. I wanted to tap into my home rig (packing a 4090 and a 3090) which I currently use for Local LLMs and want to utilize it for this.

The big question: How do I expose my home server to my cloud environment without throwing security out the window? 

The answer: Docker and Tailscale. Below is how I set it up, keeping GPU usage cheaper and my machine safe from unwanted traffic.

Using Tailscale for Secure Networking

Tailscale provides a simple, secure VPN solution built on WireGuard. It allows devices to connect directly and securely, forming a mesh network that's perfect for securely linking to machines. In this case two containers

• I have my actual service deployed in Digital Ocean Droplet with Docker Compose
• I want to expose service from my home GPU server only for docling, wrapped in a different service.
• The home GPU server shouldn't open any ports to public internet but rather my container in Droplet should connect to container in my home server using a VPN (Virtual Private Network).

Setup Tailscale

Tailscale for various tasks, but isolation is essential to ensure each device only accesses what it should. Tailscale’s Access Controls (ACLs) help you achieve this. By combining tags and ACLs, you can ensure that in this scenario, two containers only have access to each other—and not the rest of your network—effectively containing potential security issues within those containers.

ACLs function by letting you specify explicit rules for what can communicate with what, while anything not covered remains off-limits.

The following rules allow containers tagged as your-svc-tag to communicate only with each other, unless other rules permit additional connections

//Allow traffic from devices with tag:your-svc-tag devices with same tag
{
    "action": "accept",
    "src":    ["tag:your-svc-tag"],
    "dst":    ["tag:your-svc-tag:*"],
},

Setup Docker with Tailscale

The way your Docker container works with Tailscale is that you have container for Tailscale that is handling the networking and then your app container uses the network through the Tailscale container.

Following is the example of docker-compose.yml file that I use for my home machine which has the GPU that I want to expose to my Digital Ocean Droplet VM

version: "3.8"

services:

  app:
    container_name: your-service-name
    # ... (other properties)
    environment:
      # ... (other properties)
      - NVIDIA_VISIBLE_DEVICES=all
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              capabilities: [gpu]
              device_ids: ['0']
    depends_on:
      # ... (other properties)
      - tailscale
    restart: unless-stopped

  tailscale:
    image: tailscale/tailscale:latest
    hostname: name-of-service-host
    environment:
      - TS_AUTHKEY=${TS_AUTHKEY}
      - TS_HOSTNAME=name-of-service-host
      - TS_EXTRA_ARGS=--advertise-tags=tag:your-svc-tag
      - TS_SERVE_CONFIG=/etc/tailscale.config.json
    volumes:
      - /var/lib:/var/lib
      - /dev/net/tun:/dev/net/tun
      - ./config/tailscale.config.json:/etc/tailscale.config.json
    cap_add:
      - NET_ADMIN
      - NET_RAW
      - SYS_MODULE
    restart: unless-stopped

Here some of the key things are

• TS_AUTHKEY: This is auth key you generated for this purpose. We'll talk more on how to generate one.
• TS_HOSTNAME: This is what you will use to connect to this node, also what shows up in Tailscale Dashboard
• TS_EXTRA_ARGS: It indicates that when Tailscale daemon is started then it advertises the tag that you created for this purpose. This is important as that is what will allow ACLs to work properly.
• TS_SERVE_CONFIG: This is the Tailscale config, which will allow you to serve selected ports so that the Droplet can access it. You will be connecting to this service using name-of-service-host:PORT.

Tailscale config

The tailscale.config.json will be dependent upon how your service is setup, but for basic 8080 hosted service you can use something like following

{
  "TCP": {
    "8080": {
      "HTTP": true
    }
  },
  "Web": {
    "${TS_CERT_DOMAIN}:8080": {
      "Handlers": {
        "/": {
          "Proxy": "http://app:8080"
        }
      }
    }
  },
  "AllowFunnel": {
    "${TS_CERT_DOMAIN}:8080": false
  }
}

Generate Auth Key

You can go to Tailscale Settings and then click on "Generate Auth Key", to see the following dialog where you need to select various settings

• Reusable: As your container can start again, I believe this is needed so auth key can be reused when you deploy an updated container
• Ephemeral: If you restart/create new container and old one goes offline then it gets auto-removed
• Pre-approved: This is up to you depending if you want these containers to automatically connect or need your explicit approval whenever you restart it.
• Tags: Select the 'your-svc-tag' here
• Expiration: Set whatever you are comfortable with but be aware that if you start a new container after the expiry time, Tailscale will fail to connect and you would need to generate a new auth key.

Host discovery

Although you’ve set a hostname for your service, when containers restart, they connect to Tailscale with an incrementing suffix. For instance, if your hostname was foobar-hostname, it might register as foobar-hostname-1, and so on.

To identify the correct hostname to connect to, you need some form of host discovery. One approach is using Tailscale APIs to find an online node whose hostname begins with the given prefix.

import requests
from datetime import datetime, timezone
from typing import Optional, List

class TailscaleDiscovery:
    def __init__(self, api_key: str):
        self.api_key = api_key

    def find_node(self, name_prefix: str, max_idle_seconds: int = 300) -> Optional[str]:
        """
        Find the most recently active Tailscale node matching the given prefix.
        
        Args:
            name_prefix: Prefix to match node names against
            max_idle_seconds: Maximum seconds since node was last seen (default: 300)
        
        Returns:
            str: Hostname of the matched node, or None if no match found
        """
        try:
            # Fetch all nodes
            response = requests.get(
                "https://api.tailscale.com/api/v2/tailnet/-/devices",
                headers={"Authorization": f"Bearer {self.api_key}"}
            )
            response.raise_for_status()
            nodes = response.json()["devices"]

            # Filter and sort matching nodes
            matching_nodes = []
            now = datetime.now(timezone.utc)
            
            for node in nodes:
                # Skip nodes that don't match prefix
                if not node["name"].startswith(name_prefix):
                    continue
                    
                # Check if node is recently active
                last_seen = datetime.fromisoformat(node['lastSeen'].replace('Z', '+00:00'))
                if (now - last_seen).total_seconds() > max_idle_seconds:
                    continue
                    
                matching_nodes.append(node)

            if not matching_nodes:
                return None

            # Return hostname of most recently seen node
            return sorted(
                matching_nodes,
                key=lambda n: n['lastSeen'],
                reverse=True
            )[0]['hostname']

        except Exception:
            return None

In this code, you retrieve the list of devices, filter those matching the provided prefix, and check when each device was last seen (i.e., the most recent time it was online).

You can adapt this further to cache domain information. If you have multiple machines, you could implement a round-robin approach to distribute requests. Once you identify the correct Tailscale hostname, you can simply call your service.

Some issue I ran into

when running Docker inside it. I couldn’t get the Tailscale container on the Droplet to function properly—particularly for DNS resolution. Instead, I ended up configuring Tailscale directly on the Droplet (with the Auth Key, Tags, etc.) and then set my container to use the host network:

services:
  app:
    # ...(othe properties)
    network_mode: host

DeepSeek-R1 has created quite a stir in the AI world, with headlines all over the place, primarily focusing on the low cost aspects of it to create a competitive model that butts head with o1 reasoning models from OpenAI.
I was more curious on the Reinforcement Learning aspects of it which I feel is step change for how these models are built. Some private labs may have already discovered this but its good to have these techniques now in the open and also proving their usefulness.

Current challenges

For Supervised Fine Tuning, you need a decent amount good quality data which either needs to be annotated by humans or through synthetic data generation where you would then need to filter between good vs bad generated data. This is a cumbersome process and also costly. For training reasoning, you would need data that would contain the reasoning step and there is not a whole lot of data on internet that does that.

One of the earlier emergent LLMs behavior was the Chain of Thought, which essentially was that when you ask LLM to "think step by step", you would get reasoning steps and generally a better answer. DeepSeek-R1 uses this technique to train a model to make reasoning part of its response, out of the box.

Reinforcement learning

The mechanism they used was Reinforcement Learning (RL), which refers to training the model through trial-and-error by rewarding outputs that contain correct, verifiable answers and reasoning steps. This iterative process helped the LLM systematically improve its reasoning capabilities as it learned from the reward signals.

There have been other approaches to train LLMs using reinforcement learning, but till now many of them were costly; either requiring more data, or more complex training, leading to costly training. What DeepSeek has done is make it cheaper and scalable to use RL for LLM training and avoiding Supervised Fine Tuning (SFT).

The main part of the training that help achieve this was Group Relative Policy Optimization. Its a reinforcement learning method which avoids a separate critic model (in previous research) by sampling multiple answers for a question and using group scores to estimate the baseline, which lowers the resources needed for training.

DeepSeek-R1-Zero

This is where they used DeepSeek-V3-Base as the base model and applied reinforcement learning to enhance reasoning capabilities. While successful, they noticed the model's outputs had readability issues and occasionally mixed different languages.

Training Process

The reinforcement learning approach relied on multiple reward functions. Reward functions take the output and then give some +ve, 0, -ve reward.

1. Rule-based Rewards: They used questions with verifiable answers (code, math, logic) from their dataset. The model had to generate both reasoning steps and the final answer. Accurate answers received rewards while incorrect don't.
2. Format Rewards: The model received additional rewards for following specific output formats, using <think>...</answer> tags followed by <answer>...</answer> tags.

During each reinforcement cycle, the model:

• Generated multiple outputs for each question
• Received rewards based on correctness and format
• Used Group Reward Policy Optimization (GRPO) to incrementally improve towards better-rewarded outputs

Emergent Behaviors

What's fascinating is that they only provided a simple system prompt asking for reasoning in a specific format, without dictating how to reason. During training, the model developed self-reflection, self-verification capabilities, often re-evaluating and adjusting its reasoning mid-process. It also learned to reason for longer as the training went on.

This demonstrates the power of reinforcement learning - you don't explicitly teach how to reason, but rather let the model learn through rewards. This approach could potentially be applied to train models for various other capabilities, as long as you can create verifiable rewards. While this opens new possibilities for model development with less data dependency, some initial bootstrap data remains important (as we'll discuss later).

DeepSeek R1

Because of some of the issues with R1 e.g. readability, mixing of language, likely degradation on tasks that weren't specifically had verifiable rewards e.g. creative writing etc. they trained R1 to improve the overall usability of the model.

R1 was trained in multiple phases

Phase 1 - Cold start with SFT

Taking DeepSeek-V3-Base, used Supervised Fine Tuning on cold-start data that was collected from R1-Zero checkpoint. The data was likely handpicked to make sure it is of high quality.

Phase 2 - Reasoning oriented RL

Similar to R1-Zero, this phase used RL to train on reasoning related tasks on verifiable questions/answers. They did add another reward in the process which was Language Consistency Reward. This reward made sure the model avoided language mixing. Later they ran ablation experiments, where they remove this reward to understand its impact. What they saw was that Language Consistency Reward did decrease model performance but as it led to more human readable model, so its worth having.

Phase 3 - Rejection sampling and SFT

After the Phase 2 is completed, from that model checkpoint, they collect more data for Supervised Fine Tuning for subsequent round. But here they not only have reasoning oriented data but also data for other domains e.g. creative writing, role-playing, general tasks to make sure the model is well rounded. Gathering good data is important.

For reasoning related data, they generate data containing accurate answers, doesn't have mixed-languages or long paragraph etc. At the end of process they have 600K reasoning related training samples.

For non-reasoning data, they use data they had for SFT in the DeepSeek-V3 model. The use DeepSeek-V3 to also generate reasoning data for such tasks, but also exclude it for tasks that don't require reasoning so that model also learns to avoid reasoning when its not needed.

Using all this data they train the DeepSeek-V3-Base with this curated dataset of 800K samples.

Phase 4 - RL for all scenarios

In this final state a round of Reinforcement Learning is done, but not only on reasoning data but also on other data where the reward is based on human preference. There is also safety related (harmlessness) evaluation done during this phase.

At the end of all these phases we get the DeepSeek-R1 model and my vibe check on that model has been pretty good.

Distillation

Lastly they use the R1 model to generate 800K sample data and use Supervised Fine Turning on various small language models e.g. Llama-3.3-70b-Instruct, Qwen2.5-32b etc. and observe that those models overall improve in complex tasks requiring reasoning.

The other interesting thing they also observe is that directly using RL on those models similar to R1-Zero, doesn't improve those models as much as distillation from larger model does.

Whats likely to happen next

The year 2025 is going to be interesting, we are going to see much more research in direction of RL, many more distilled small models with reasoning capabilities that will be SOTA. The test time scaling will be something that Open source/weights models will also be pushing in direction off. Likely more (verifiable) domain specific small models may become easier to train due to RL. Anything that you can verify relatively cheaply in training process, you are likely to see much improvements in models related to those.

We are already seeing various OpenSource capability to train using RL (e.g. Unsloth). Here is my attempt to train reasoning with emojis in each step using Unsloth 😄

<reasoning>
Her 200 sheep each produced 10 pounds of wool, so the total amount of wool is 200 * 10 = 2000 pounds 🐑.
She sold one pound of wool for $20, so the total revenue is 2000 * $20 =$40,000 💸.
The cost to her was $2000 💸.
So, her profit is $40,000 - $2000 = $38,000 🤑.
</reasoning>
<answer>
$38000
</answer>

Its certainly going to be exciting 2025 for open source models.

Disclaimer: I'm no ML/AI expert, but a curious engineer who has been learning more about the inner workings of the LLMs to try to understand better the new techniques (when it comes to using them) and also how these models are becoming better. If you find something incorrect in my understanding please leave a comment.

I transitioned to the role of Engineering Manager approximately 5 years ago, since then I haven't been programming in my day job but the itch to do so has always been there. So I continue to work on side projects to not lose touch and continue to hone my skills.

Because my time has always been limited, progress on side projects had been slow in the past, and many remained unfinished as life's events caused a loss of momentum, making them harder to resume. However, in the last year (2024), I have been very productive with my side projects, quickly building the tools or projects I need and deploying them for others to use—in other words, finishing the v1 of each project.

A few examples of what I've built are

• jsonplayground.com - JSON formatter but also in browser JQ using WASM so no data leaves the machine.

• webtomarkdown.com - As I often feel the need of converting files to Markdown, or parts of website to Markdown for passing in as context to LLMs. I'm currently building this tool to solve that problem.

• Face lift for my soaring club page Evergreen Soaring where I volunteer (not deployed yet on official website).

• A Chrome Browser Extension to automate parts of public messages we receive at my soaring club.
• fitinterval.com - Interval timer for workouts

LLMs in general have been immense booster for my productivity when it comes to side projects and more specifically the Cursor IDE has been a great editor to use these LLMs for coding.

In this blog I'll go over what my high level flow looks like for greenfield projects and I hope that may help you. I do want to acknowledge that these tools are good in certain cases but may annoy you (waste time) in other areas, you just need to use them to figure out where specifically it's useful for you.

I have a nice habit tracker that I would like to replicate as a website, but all data stored locally, so let's use that as an example of what to create here.

Start with a spec

I use the o1 ChatGPT to first get my application specification more refined. The reason I do that is so that it helps me scope the problem and also the spec I get at the end, I use it in further stages of bootstrapping the code. You can try to write the spec yourself, but I feel that going from few sentences to more detailed spec through ChatGPT o1 has been very useful in saving me time. I also ask it to further probe me with questions to further refine it.

Following the prompt I start with.

I’m want to build a website for habit tracking where user sees columns of months and each row being a date. They can simply select to indicate a day where they continued with the habit. It should store all that on local machine. Ask me more questions to refine the idea.

It asks me bunch of questions which I answer, but then it continues asking me more questions. At some point, where you feel there is enough details, you should explicitly ask it to create a spec with the details that will allow another person/AI to build application. I also specify the technology I would prefer to use as thats what I'm familiar with most.

Answer those questions for me reasonably and create a spec that I can give to a person or another AI to help create the website. Make sure to have the details of project, user experience, technical details. I want to use typescript, react, tailwind css.

You can read the whole chat here: Habit Tracking Website Plan

Now store that spec in SPEC.md in a folder where your project will be. We will continue to refer back to it when needed.

Bootstrap project

I use Vite to bootstrap my project. This allows me to setup all the necessary tooling in a consistent manner.
In the directory of project I run npm create vite@latest . which will ask me question about which UX framework and Language to use. Once I have the project and SPEC.md in that project I use the Cursor Agent to create the initial code.

You can go to Composer > Select Agent > Added SPEC.md in the context and ask it to implement it.

This will go over your code, setup tailwind, update few files to create the initial version.

This is what the initial version looked like. Not exactly what I was looking for (skeuomorphic design) but close enough in structure that I can iterate over it.

There is also some bug in it, where clicking on the button doesn't change the state. But overall, this puts us into a good starting point, it created the overall UX layout I expected, stored data in local storage, has the right export feature for Markdown. All of it in just order of minutes instead of hours.

P.S: Sometime I also use v0.dev to bootstrap the UX aspect of the project. That tools allows quicker iteration on the UX aspects.

Small iterations

You don't want to one-shot everything i.e. ask it to do multiple complicated tasks in one go. That can sometime work but can lead to issues and makes it harder (and slower as it will regenerate bunch of code that it doesn't need to change) to iterate. Follow a divide and conquer approach, i.e. split your feature into smaller tasks and iterate over them using the Chat/Composer.

Now first let's fix the bug and also change the UX. In my spec conversation with o1, I ask it to create a spec for UX focused more on skeuomorphic aspects of it. Then I use the Cursor Composer to update the code. I select the o1 model in this case.

Update @App.tsx @MonthColumn.tsx @MonthColumn.css @App.css to improve the UX, also fix the issue where the state isn't being changed when I click the button.

{PASTE THE UX SPEC}

Here is what it looks like now. So it fixed the bug, and also updated the UX to have some more Led like behavior with depth, some shadows etc. It still look horrible but we will further iterate on that.

In the next iteration I gave the above screenshot (yes cursor can also use images for context) in Chat mode and first asked it to describe the details the button and then asked it to make necessary changes to replicate that. After couple of more iterations.

After few back and forth, I have the experience which looks good enough for the demo here.

Now finally I need to setup deployment using GitHub actions, so whenever I check-in to main, it builds and deploys to GitHub pages. I already had a workflow in another of my repository that I wanted it to use as context and make specific changes to build this project. The good thing about Cursor is that you can also provide context by adding a link, so either its some existing code, some documentation, it can be passed to LLM for context. In my experience providing relevant context generally allows it to output better code and avoid hallucinations.

Similar to @https://raw.githubusercontent.com/zabirauf/evergreensoaring-modern-web/refs/heads/main/.github/workflows/deploy.yml create a deployment to github pages and also make sure to npm install and npm run build (which puts it in dist folder). The dist is what needs to be deployed

Overall tips

1. Use LLM to hash out the details of the project and store it for further context
2. Use a tool or open-source template to bootstrap your project to setup all the necessary toolings and following a manageable project pattern.
3. Leverage Cursor Composer (agent mode) to bootstrap the project
4. Use mix of o1 and claude-3.5-sonnet. Generally I use o1 where broad stokes are needed e.g. 1st draft of a feature and then use Claude-3.5-Sonnet to further iterate on it. But I'm using Claude-3.5-sonnet approx. 80% of times.
5. Select the right mode e.g. Chat, Composer (normal), Composer (agent).
   I use Chat, when I need back and forth and know exactly where changes will be and want to see the changes before applying.
   I use Composer (normal) when I need multi-file changes e.g. new feature.
   I don't use Composer (agent) often enough yet. Composer (agent) can run commands in terminal, lint code, re-iterate etc, but going back to the principle of small iterations, I try to scope things to what I can review easily and add.
6. Provide relevant context as much as possible e.g. specific files you want changed, specific docs (links), or submit with codebase option in chat when you want it to search for relevant context.
7. Store markdown files relevant to your project so you can add those as context e.g. SPEC.md, documentation from website that you often get back from (plugging https://webtomarkdown.com for converting a website documentation to Markdown and storing it 😄)
8. Create and use .cursorrules file in your project directory for instructions that you want it to take in prompts, e.g. if you see it always using some library you don't want then add it to .cursorrules, specific technology that you want it to user in code e.g. Tailwind, certain component library e.g. Shadcn etc. This allows you to start nudging it in direction you want for most of your prompts.
9. Always make sure that you understand the code at high level so you don't land in a space where eventually it's such a messy code that it becomes hard for you to debug when LLMs can't find issues for you. My tip is to continue to split stuff into manageable pieces (hint, you can use LLMs to do it from time to time).

Closing remarks

I hope this has been helpful, and that you can start finishing the first versions of your projects and deploying them. By turning unfinished projects into completed and deployed ones, you can continue to build momentum even when you take small breaks. This approach allows you to gradually add more to your projects while keeping them manageable. I believe this also helps keep me motivated, as I get to see progress more quickly on what I want to deliver.

When setting up a NixOS system, we often want to store secrets in environment variables for various tools to use. For example, you might want to use the llm CLI tool, which requires setting the OPENAI_API_KEY environment variable.

However, if you're using declarative configuration files or flakes to set up your NixOS system (which is a great practice!), you can't simply update those files with the secret. That would be insecure, especially if you're tracking your configurations in Git and pushing to a public repository.

Enter SOPS

To manage secrets on machines, we can use a tool called SOPS (Secrets OPerationS). SOPS allows you to manage secrets by encrypting them when storing them in a file. You use the tool to edit the secrets, and when you save, it writes them in an encrypted manner.

In NixOS, we have sops-nix, which helps manage SOPS-based secrets using configuration. Here's how it works: when SOPS secrets are loaded from a file at runtime, each secret is stored in a different file. This means at runtime, you can read from those files to get the secret and export it as an environment variable.

Setting Up SOPS with Age

We'll use Age as the encryption backend for SOPS.

Install Age and SOPS

In your home manager where you manage packages make sure to add age and sops and rebuilt to install them.

Generate an Age Key

First, create a directory for SOPS configuration and generate an Age key pair:

mkdir -p ~/.config/sops/age
age-keygen -o ~/.config/sops/age/keys.txt

The public key is derived from the private key:

age-keygen -y ~/.config/sops/age/keys.txt

Copy the output public key; you'll need it to encrypt your secrets.

Create the .sops.yaml Configuration File

Create a .sops.yaml file in your project's root directory to define encryption rules and specify the public key:

   keys:
  - &host_hostname <YOUR PUBLIC KEY>
creation_rules:
  - path_regex: secrets.yaml$
    key_groups:
    - age:
      - *host_hostname

Replace <YOUR_PUBLIC_KEY> with the public key you obtained earlier. And replace hostname with your machine host-name. This configuration tells SOPS to use Age for encrypting any file named secrets.yaml using the specified public key.

Create and Encrypt the Secrets File

Use nix-shell with SOPS available to run creating the secrets.yaml file. Make sure to run it from the same directory where .sops.yaml exist. It will open the editor that is configured using EDITOR in your shell.

nix-shell -p sops --run "sops secrets.yaml"

Add your secrets to the file:

openai_api_key: 'YOUR_OPENAI_API_KEY'

Save and exit the editor. SOPS will encrypt the file upon saving, and secrets.yaml will now contain encrypted data along with other metadata.

Integrate SOPS with NixOS Using sops-nix

Update Your flake.nix

Add sops-nix to your Nix flakes:

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-23.05";
    sops-nix.url = "github:Mic92/sops-nix";
    # other inputs...
  };

  outputs = { self, nixpkgs, sops-nix, ... }:
    let
      system = "x86_64-linux";
      pkgs = import nixpkgs { inherit system; };
    in
    {
      nixosConfigurations.mySystem = pkgs.lib.nixosSystem {
        inherit system;
        modules = [
          ./configuration.nix
          sops-nix.nixosModules.sops
          # other modules...
          home-manager.nixosModules.home-manager
          {
            # other config...
            home-manager.sharedModules = [
              sops-nix.homeManagerModules.sops
            ];
          }
        ];
      };
    };
}

Adding sops-nix to the Inputs:

inputs = {
  nixpkgs.url = "github:NixOS/nixpkgs/nixos-23.05";
  sops-nix.url = "github:Mic92/sops-nix";
  # other inputs...
};

This line adds the sops-nix input, fetching the sops-nix repository from GitHub. It provides the necessary modules to integrate SOPS (Secrets OPerationS) into your NixOS system.

Including the SOPS NixOS Module:

modules = [
  ./configuration.nix
  sops-nix.nixosModules.sops
  # other modules...
];

By adding sops-nix.nixosModules.sops to the modules list, you're integrating the SOPS module into your NixOS system configuration. This enables system-wide management of encrypted secrets.

Integrating SOPS with Home Manager:

home-manager.nixosModules.home-manager
{
  # other config...
  home-manager.sharedModules = [
    sops-nix.homeManagerModules.sops
  ];
}

This section includes the SOPS module in Home Manager's shared modules by adding sops-nix.homeManagerModules.sops. It allows you to manage user-specific secrets through SOPS within your home environment configurations.

Configure SOPS in home.nix

In your NixOS configuration, set up SOPS:

{
  sops = {
    age.keyFile = "/home/<your username>/.config/sops/age/keys.txt"; # must have no password!

    defaultSopsFile = ./secrets.yaml;
    defaultSymlinkPath = "/run/user/1000/secrets";
    defaultSecretsMountPoint = "/run/user/1000/secrets.d";

    secrets.openai_api_key = {
      # sopsFile = ./secrets.yml.enc; # optionally define per-secret files
      path = "${config.sops.defaultSymlinkPath}/openai_api_key";
    };
  };
}

This configuration tells NixOS to use the encrypted secrets.yaml file and makes the openai_api_key available at the specified path. Whenever you add a new secret, you should update the config here with that secret info as well.

Accessing the Secret as an Environment Variable

In your config you can use $(cat ${config.sops.secrets.openai_api_key.path}) to get the secret. For example I configure my zsh file to export the OPENAI_API_KEY.

 programs.zsh = {
    initExtra = ''
      # other config...
      export OPENAI_API_KEY=$(cat ${config.sops.secrets.openai_api_key.path})
    '';
  };

Reload Configuration

After updating your configurations, rebuild your NixOS system:

sudo nixos-rebuild switch

Now, your OPENAI_API_KEY is securely stored and accessible as an environment variable without exposing the key in your configuration files.

Conclusion

By integrating SOPS with NixOS, you maintain a declarative and secure configuration while managing sensitive secrets. This setup ensures that API keys and other confidential data remain encrypted in your repositories and are only decrypted on your local machine during runtime.

Remember to always be cautious when dealing with secrets and never commit unencrypted secrets to your repository!

References

• Framework and NixOS - Sops-nix Secrets Management
• GitHub Mic92/sops-nix

Ollama is a powerful and user friendly tool for running and managing large language models (LLMs) locally. But not all latest models maybe available on Ollama registry to pull and use. The fastest way maybe to directly download the GGUF model from Hugging Face. In this article, we'll explore how to install a custom Hugging Face GGUF model using Ollama, enabling you to try out latest models as soon as they are available.

I use Open-WebUI as the interface for Ollama, and all these instructions would allow you to use the model from Open-WebUI as well.

1. Download the Model

First, we need to acquire the GGUF model from Hugging Face. We'll use the Hugging Face CLI for this:

huggingface-cli download --help bartowski/Reflection-Llama-3.1-70B-GGUF Reflection-Llama-3.1-70B-Q4_K_S.gguf

This command downloads the specified GGUF model, which in this case is a fine-tuned version of LLaMa 3.1.

2. Create a Modelfile

Modelfile is the blueprint that Ollama uses to create and run models. Since we're working with a LLaMa 3.1 variant, we can base our Modelfile on an existing one:

ollama show --modelfile llama3.1:70b-instruct-q4_0 >>  Modelfile

This command generates a Modelfile based on the llama3.1 model specifications which I already had locally pulled.

If you don't have example of existing Modelfile to reuse then you would need to figure it out from the Hugging-Face page for the model and then create one.

3. Update the Modelfile

Now, we need to modify the Modelfile to point to our downloaded GGUF model. Open the Modelfile in a text editor and update the FROM line with the path to the downloaded model. The Hugging Face CLI will have printed this path at the end of the download process.

4. Create the Model in Ollama

Finally, we'll use Ollama to create our custom model:

ollama create mattshumer/Reflection-Llama-3.1-70B:Q4_K_S -f Modelfile

This command processes the Modelfile and copies the model to Ollama's storage, typically located at /usr/share/ollama/.ollama. Here mattshumer/Reflection-Llama-3.1-70B:Q4_K_S is the name of the model that I will use in Ollama, you can name it whatever you want.

If you want to delete the hugging face cached model so you don't use double the storage then you can run the following which will start a Terminal UI for you to select which models to delete.

huggingface-cli delete-cache

By following these steps, you've successfully installed a custom Hugging Face GGUF model using Ollama and in Open-WebUI.

In this post I'll go over on approach for onboarding new early hires remotely that I've formed from my experience. Hope it helps you and your team in the process.

I transitioned to Engineering Management role, around 2 and half years ago. During that time most of it was spent managing the team remotely but I don't claim to be an expert at it but rather student as I go along and learn.

Before last year, most of the team I managed were based in same city though working from home due to COVID. These were the people I've met and worked with in person. My team recently hired multiple engineers (early in SWE career), some in different state in USA and others in LATAM.

It was an interesting challenge onboarding them, so they become productive, make them feel part of the team, get them energized around the mission and work they are doing. I'm glad to see them now contributing to our product and team. Few things you should know about it

1. It will take time commitment both you as their manager, onboarding buddy/mentor and other members of the team.
2. There are managerial challenges you will have to understand due to different countries and laws (equipment, benefits, holidays etc.).
3. Building connection with the team members and between them and team happens gradually but only with conscious effort.

Here are learnings from my experience and suggestions on what I currently think provides a good onboarding model

Homework before start date

You goal should be that in the first week have the new team member setup their machines and complete their first check-in. For this you have to do some homework before their first day

1. Make sure all the steps to setup their machine for development, cloning repo etc. are written down (more on writing later).
2. You need to pick an onboarding buddy/mentor but it shouldn't be random member from team. Think about what area you want the new engineer to continue working in about 3-6 months. Pick the onboarding buddy who owns that area. That will help build a relationship between them from the get-go and they will be more open to discuss issues when they work on complex projects.
3. Allocate time in onboarding buddy/mentor sprint for this. Make mentoring an explicit goal for them, this will ensure they don't take it as an extra ask alongside their regular sprint.
4. Ask the onboarding buddy to find couple of easy bugs that would be suitable for someone new to tech stack and code.
5. Create a PowerPoint/Email and ask your team members to share some details about themselves. For example, what they work on, preferred communication mode, fun fact about themselves, picture etc. This will be a great way for the new person to know the team, understand their preferred times and communication mechanism (sync/async). Send it on their starting day as a first thing.
6. Before they join, send them a welcoming email and do include your contact so they can reach to you if Teams/Zoom doesn't work.

On their first week

1. Have a 1-1 with them on the start of their first day. You can do introductions, share what team does, how they should get started etc. Basically sharing all the homework you did to set them up for a good first week.
2. Schedule your regular 1-1, at minimum once per week. If you generally do bi-weekly cadence, still have once per week for first few weeks and then later you can transition to bi-weekly cadence. It's essential they get face time with you to discuss their issues, understand more about the work, team culture and company.
3. In the first week, would highly recommend doing quick check-in, at least once in 2 days. Have their onboarding buddy setup a daily (preferable) 1-1 with them. People can be hesitant to reach out for help so having face time with their onboarding buddy will ensure they are not stuck.

Growing responsibilities and support

I hope with above instructions they were able to get in few quick wins (check-ins) and feel confident navigating parts of the codebase, tools and processes.

Initially you would give them bug level items and smaller work so they build confidence in the first month. After that start forming a plan to have them work on a certain feature or larger change in the area of work you determined earlier.

By this time they will be comfortable working with their mentor who is also the owner of the area where they'll be working. Make sure that you continue to sync with them in 1-1, understanding how they are progressing.

In the beginning there can be unnecessary stress exacerbated by being remote. They may think they are underperforming as they don't have anything to gauge it. Understanding how they perceive about their own progress; you can help calm their nerves. If they are doing well, appreciate and communicate that. If they are struggling reach out to understand how you and team can support them.

Culture of writing

Our team previously had some knowledge written down but most of it was tribal knowledge passed during meetings, conversations etc. In a world where the whole team is in same space, this is not optimal but the fact that people are accessible puts a band aid on this problem.

In all remote or hybrid situation that band aid is ripped out so now you have to fix the problem. The solution is simple, write things down. Execution takes repeated effort and reminder. Following are some of the things that you should have written down. I don't claim my team or I are 100% done with it but we are improving and that's the goal.

1. How to setup machine
2. Tech stack used and primer on it
3. Architecture of the system
4. Good patterns to use
5. Various tools at developers disposal and how to use them
6. Culture (writing, collaboration, experimenting, accountability etc.)
7. Team meetings and processes (what various team meetings are, expectation from attendees etc.)
8. Dev specs (one of the real benefits of it is when there are many approaches why a particular decision was made)

You as a manager should be promoting things to be written down. Whenever you hear something that would be beneficial for others, ask the person to write it down in team wiki.

Culture of sharing

As we work remotely, sharing can become harder as there is no hallway conversation to ask people what they are working on, no in person gatherings to demo stuff.

You need to have a way for people to share technical learnings and sharing about the work they do. This could be newsletter, Teams/Slack channel or a weekly meeting but there should be a platform where it's expected to share and celebrate the work being done. This platform shouldn't be restricted to your immediate team but a broader organization (not more than approx. 50 people).

• This helps in recognizing the work the new team members are doing.
• See what others are working on. This helps make connections which will be beneficial when working on more broader projects.
• It creates energy by seeing how the team is moving towards a goal and how their contributions are part of the bigger whole.
• Sharing technical knowledge e.g. things that didn't go well, new techniques learned etc.

I recently built findvaccinefor.me which gets data from WA state sources such as vaccinelocator.doh.wa.gov and prepmod.doh.wa.gov and presents it in a more convenient way (IMO). I built this over few weekends as a tool for me to see where I can get vaccine shots and in the process maybe help few other folks like me. It certainly doesn't solve all the problems.

Professionally I work on a full stack product and overtime have gotten some sense of what a good or bad user experience feels like (at least I would like to think that). The thing I generally try to focus on is friction in user flows. Each time we make something slightly harder the funnel of users completing the flow becomes narrower.

I don't have any data but my intuition is that existing vaccine scheduling systems introduce so much problems that it might result in actual harm by people delaying vaccination.

Lets go over some of the gripes I have with existing tools and why user experience is very essential even for a mundane thing as scheduling an appointment.

Problem 1: Finding clinic

In the age where everyone uses some form of maps e.g. Google maps, Apple maps it's hard to parse through information based on zipcode and how many miles away it is. I believe majority of us are habituated to use maps for searching for a location and understanding it's closeness and convenience of getting there.

Why is it showing me dates all the way back to 2016. It should only show the dates in future, I thought that would be obvious. But it should also filter out dates for which there are no appointments available. As a user looking for vaccine, I'm only interested in dates which have an appointment available so I can pick the most convenient one out of them. Currently user has to try out multiple dates and hope there is an appointment available or browse through bunch of listing to find the right date where vaccine is also available.

Hmmm, why show me a place where there is no appointment available. What are they expecting me to do with that info? This just adds to the friction as I parse the list. Though I'm glad prepmod.doh.wa.gov shows the map but its not much useful when I can't see where the location is in relation to me.

Problem 2: Different scheduling websites

Can you spot the problem in above picture? Yep, you guessed right. Each clinic has a different scheduling system, some even points to a Facebook page. As a user everytime you click for scheduling, you are in for a surprise and have to go through the pain of understanding each system.

I believe prepmod.doh.wa.gov tries to solve that problem but it only has few bigger vaccination sties and most of the places around you would likely have their own system. These systems make you fill a form before you can even see the vaccine is available at a time that is convenient for you. This results in a lot of time spent in filling forms unnecessarily to realize they don't have it.

Instead, please show me what's available and when, only when I select they can go through the process of filling all the forms they want.

I understand why we currently have such a disconnected system, likely because each provider wants to do their own thing. I think a world wide pandemic requires a response which is more national or at least at a state level even for digital experiences, otherwise I think we run a risk of a slower vaccination pace and likely more economic hit due to it.

Kudos

I certianly loved how the vaccinelocator.doh.wa.gov provided a lot of accessibility features.

Its also nice to see that all these websites are dynamic and work well on mobile phone.

This is not to dis the people helping build it as I tend to believe they probably did what they could with the resources they had. But it's just a criticism of the general laissez faire approach to building digital experience for the goverment and not investing much time and money on it.

Jupyter notebooks in general have been really great for literate programming and exploratory learning. Google Colab allows you to directly create a Notebook and run algorithms on GPU (if you are doing Machine Learning) instead of having to deal with Jupyter or driver installations etc.

While I start to use Notebooks, one thing I've missed from the regular dev environment in an IDE is a debugger. Yes, you can always use print() as a quick substitute for debugger but sometimes you need to explore the problem more in the state it occurred, learn how the program is being executed etc. In those cases debuggers are super handy.

Gladly there is a way to debug code within Notebook albeit not as convenient as IDE but still good enough. Python already has pdb which is a command line based debugger but for Jupyter there is ipdb which is similar to pdb but for Jupyter notebooks. This debugger doesn't have any visual aspect to it as IDE do, but they allow you to perform similar capabilites such as "Continue", "Step into functions", evaluate expressions, jump to line etc.

Lets get into how to use ipdb in Google Colab.

ipdb in Google Colab

As ipdb is not packaged with Python itself so you need to install the package and import it.

!pip install -Uqq ipdb
import ipdb

Debug when hitting Exception

You can use the follow magic command in your Notebook to turn on debugging, so that whenever your code hits into an Exception, the code will pause and the debugger view will be opened for you to explore what caused the exception

%pdb on

In this example, we raised an exception and as you see that the debugger automatically starts when you hit an exception and I was able to inspect the value of i in that state. This is helpful when you don't know why an exception is occuring so you can pause at that point, inspect values and see what went wrong.

If you want to turn off this then just use %pdb off.

Setting breakpoints

If you have a complicated piece of code which isn't working as expected for certain inputs, you can add breakpoints in the code so when you execute the code, it will pause at that breakpoint.  You can inspect what's going on or even run the code step by step to see what each line of code is doing.

Adding a breakpoint is as simple as adding ipdb.set_trace(). It will cause the code to pause when it reaches this line and open the debugger view. This function also takes in parameter such as ipdb.set_trace(context=6), here context refers to how many lines of code should it print when it pauses so you can get the code context surrounding the line it paused at.

Here you can see the code pauses execution when it reaches the line and I can inspect the variables such as arr. In the next section we will see what you can do when you are in debugger view.

Commands for ipdb

When the debugger is started you will see a view like above. It show the code and has an arrow ----> pointed at the line where its paused at. The input field below is where you can run various commands or use it as a regular REPL to execute python code such as inspecting variable values, evaluating an expresion, setting variables etc.

The input field also allows you to enter the commands for the debugger that allows you to control the flow of execution such as stepping into a function, jumping to a certain line of code, continue till you hit another breakpoint etc. Following is the basic set of commands that I found useful. You can find a more exhaustive list on Python Docs for pdb.

Resources

• Github - ipdb
• pdb Python Docs
• Google Colab notebook from blog

Managing a big app always has its own set of problems. One problem that I've seen is managing the import paths to modules you are using. As your app becomes bigger, it makes it harder to manage these relative path. Especially if you have to move these files or folders around, refactoring can become harder.

How I like to manage a project is by creating folders with index.ts in it. I consider that folder to be its own package which exposes all its functionality through index.ts. This leads to a better interface of what a particular set of code provides and what are its internal details.

I have this very simple app that I created using the expo tool in React Native using Typescript. How the code was structured is that the main component refers to the counter component which is in this particular file in a separate folder /src/app-main/lib/index.ts so the import statement looks something like

import { Counter } from './src/app-main/lib/index.ts';

Typescript provider us a capability to add aliases for paths. If you open your tsconfig.json and then under compiler options you can add paths.

{
    "compilerOptions": {
        ...
        "paths": {
            "app-main": ["./src/app-main/lib/index.ts"],
            "app-counter": ["./src/app-counter/lib/index.ts"]
        }
    }
}

This tells Typescript how to resolve app-counter to that particular file.
This is not the complete picture. In case of React Native created using expo, Babel is the one that's actually doing the transformation, so we need to tell it about those aliases as well. Following are the changes that you have to make to babel.config.js

module.exports = function(api) {
    api.cache(true);
    return {
        ...
        plugins: [
            [
                'module-resolver',
                {
                    ...
                    alias: {
                        'app-main': './src/app-main/lib/index.ts',
                        'app-counter': './src/app-counter/lib/index.ts',
                    },
                },
            ],
        ],
    };
};

Now I can get rid of all that relative path and make the imports more cleaner e.g.

import { Counter } from 'app-counter';

Make sure you start expor using -c e.g. yarn start -c otherwise it might still use the cache and not able to resolve those new aliases.

This also make it super easy to refactor as you can change where the folder and files are placed and only need to update the aliases, rest should work fine.

Imagine a world where a software is maintaining millions of dollars worth of money, but there is an exploit which allows the hacker to take all that money away. Now imagine you can't do much about it because you can't update your software, so all you can do is wait and watch or pull the plug of the whole server.

This is a world we are living in with the software/contracts being developed for the Ethereum blockchain.

Immutability of blockchain has its advantages in making sure that things are tamper proof and the whole history of change can be seen publicly and audited. When it comes to smart contracts, immutability of the contract code has its disadvantages which makes it hard to update in case of bugs. The DAO and the Parity Wallet exploit are a good example of why smart contracts should have the capability to upgrade.

There is no de facto way of upgrading a contract. Several approaches have been developed and I'm going to discuss one of the better ones in my opinion.

Understanding how contracts are called

In Ethereum virtual machine (EVM) there are various ways by which contract A can invoke some function in contract B. Lets go over them

1. call: This calls the contract B at the given address with provided data, gas and ether. When the call is made the context is changed to this contract B context i.e. the storage used will be of the called contract. The msg.sender will be the calling contract A and not the original msg.sender.

2. callcode: This is similar to call but the only difference is that the context will be of original contract A so the storage will not be changed to that of the called contract B. This means the code in the contract B can essentially manipulate storage of the contract A. The msg.sender will be the calling contract A and not the original msg.sender.

3. delegatecall: This is similar to callcode but here the msg.sender and msg.value will be the original ones. It's as if the user called this contract B directly.

Here delegatecall seems of interest, as this can allow us to proxy a call from the user to a different contract. Now we need to build a contract that allows us to do that.

Upgradeable smart contract

We will write a smart contract that has an owner and it proxies all calls to a different contract address. The owner should be able to upgrade the contract address to which the code proxies to.

This is what that contract would look like

Let's dissect the code

Let's dissect above code. function () payable public is the fallback function. When a user calls a contract without a function call (when sending ether) or a function call that is not implemented, it all gets routed to the fallback function. The function can do whatever it wants to. As we don't know what functions our contracts will have, nor how will it change so this is the ideal place where we route all calls made by user to the required contract address.

Inside this function we have some EVM assembly, as what we want to do is not currently directly available in Solidity.

let ptr := mload(0x40)
calldatacopy(ptr, 0, calldatasize)

As the user can send data which would contain information about which function to call and what are the arguments of that function so we also need to send it to our implementation contract. In Solidity the address to free memory is stored at location 0x40. We load the data at 0x40 address and assign it name ptr. Then we call calldatacopy to copy all the data passed in by user to memory starting from ptr address.

let result := delegatecall(gas, _impl, ptr, calldatasize, 0, 0)

Here we use the delegatecall as discussed before to call the contract. The result returned is either 0 or 1.
0 means that there was some failure so we need to revert. 1 means the contract code execution was successful.

let size := returndatasize
returndatacopy(ptr, 0, size)

As the execution of code might have returned some data so we need to return it back to user, here we copy the returned data to memory starting at ptr address.

switch result
case 0 { revert(ptr, size) }
case 1 { return(ptr, size) }

Lastly we just decided whether to return or revert based on the result of delegatecall.

We inherit the contract from Ownable so the contract will have an owner. We expose updateImplementation(address) function to update the implementation and it can only be executed by the owner of contract.

Deploy an upgradeable contract

First deploy the UpgradeableContractProxy contract. Let's say we have a global calculator contract, which has list of numbers in storage and we can get the sum of all numbers in it.

Let's take that contract and deploy it. It should give us some address, now use that address as one of the parameters to call updateImplementation of the previously deployed UpgradeableContractProxy contract. Now our proxy should point to the deployed GlobalCalculator_V1 contract.

Now use the GlobalCalculator_V1 contract interface (ABI) to call the proxy contract. So if you are using web3js then you can do the following

const myGlobalCalculator = new web3.eth.Contract(GlobalCalculator_V1_JSON, 'PROXY_CONTRACT_ADDRESS');

Now when you make a function call to proxy, it will delegate the call to your global calculator contract. Let say you called few addNum(uint) to add numbers to the _nums list in storage and now it has [2, 5] as value.

Let's add a multiplication functionality in your updated contract. We create a new contract which inherits from GlobalCalculator_V1 and add that feature.

Let's deploy GlobalCalculator_V2 contract and call updateImplementation(address) of the proxy to point to this newly deployed contract address.

You can use the GlobalCalculator_V2 contract interface (ABI) to call same proxy contract which would route the call to new implementation.

const myGlobalCalculator_v2 = new web3.eth.Contract(GlobalCalculator_V2_JSON, 'PROXY_CONTRACT_ADDRESS');

You should have the ability to call getMul() now. One thing to notice is that if I call getSum it will return 7 as the _nums in storage is still the same as the old one and have value [2, 5]. So in this way our storage is maintained and we can add more functionality to our contract. The users also don't have to change the address as they always call the proxy contract which delegates it to other contracts.

Pros

1. Ability to upgrade code.
2. Ability to retain storage, no need to move data.
3. Easy to change and fix.

Cons

1. As we use fallback function to delegate so ether can't be directly sent to contract (due to not enough gas when using .transfer or .send). We need to have a specific function that needs to be called for receiving ether.
2. There is some gas cost for executing the proxy logic.
3. Your updated codes needs to be backwards compatible.
4. The update capability is controlled by one entity. This can be solved by writing a contract that uses tokens and voting mechanism to allow the community to decide whether to update or not.

If you want to look at an example project using truffle which is testing an upgradeable contract see this.

Disclaimer

Don't use the above contract for production as it's not audited for exploits. It is only provided here for educational purpose.

Reference

• Github example project
• Zepplin Ownable contract
• Zepplin OS Proxy contract