Imagine a support assistant promising an instant refund, even though the policy requires manager approval and three to five business days for processing. That’s a response worth keeping as a test. 🤔

I wanted to keep a bad response as a test I could rerun after changing the application. LLMCheck is my Python project for capturing model calls, reviewing failures, and turning them into regression cases.

Let’s capture that response, edit the generated test criteria, run a new response through the test, and inspect the verdict together. 🧪 We’ll also try answers that could fool the checker. Those checks exposed a draft-generation defect and showed that low confidence does not prevent a passing result.

The walkthrough uses LLMCheck code with a made-up refund policy and scripted responses. It runs offline. Later, we compare a literal string checker with a recorded OpenAI evaluation of twelve examples from that policy.

The judge is the model or function that checks an answer against the case’s criteria. We will inspect its explanation and reported problems alongside each verdict.

You can inspect the refund-policy cases in the explorer or run the current refund-policy example. The optional setup below reproduces the older implementation used in this walkthrough, including the defect discussed later.

What LLMCheck Actually Is 🔍

LLMCheck connects four operations: capture a call, review its output, save an approved case, and run the application against that case again. It stores captured records in SQLite and regression criteria in YAML. At the source revision used here, the instrumentation wraps the synchronous client.chat.completions.create(...) interface.

LLMCheck checks response text. You still need application tests to confirm that a database transaction committed, an advert was updated, or an external service accepted an action. It also does not provide production tracing or wrappers for every provider’s SDK.

Review the case, run the application, and inspect the judge’s reported violations. Python uses those violations to calculate the verdict.

The default judge receives the new output, approved context, and expected criteria, then returns lists of violations. Python passes the case when those lists are empty. A judge that misses a problem can therefore produce a false pass. We will look at that code below.

The repository also has an experimental feature for reusing reviewed knowledge in later calls. We will cover it after the regression walkthrough.

The offline walkthrough and its original figures use source commit 6d101ae90781b8dc06965f57313445f8878cf6d6, version 0.3.0. Inspect the historical source alongside the demo scripts. The release update and recorded live evaluation below cover later work. Use the pinned commit to reproduce the original examples, including the defect.

Release update: what changed after the walkthrough

The core is licensed under Apache 2.0. PR #3, merged as 70a6fde, fixes the draft-generation defect examined below and hardens capture, dashboard rendering and installation. The merged revision passed 58 tests in the checkout and extracted source archive, with CI passing on Python 3.10, 3.11 and 3.12.

The earlier 0.4.0 candidate, ce91167, passed 53 local tests. In the fixed generator, correction text stays in the rubric; reviewers fill in the required and forbidden phrase lists.

Open the case explorer to switch between the literal fixture and the recorded OpenAI run. Clicks display saved evidence and do not call a provider. Inspect the archived experiment scripts to reproduce the original 0.3.0 walkthrough, including its historical defect. The live run used the modified candidate before it was committed: its base Git revision and exact judge/script hashes are preserved in the release evidence.

Why This Problem Matters to Me 💭

I worked on a publishing data platform, operating and extending recommendation workflows across 37 configured production publishing sites.

That work included historical and incremental processing. Processing an existing dataset can follow a different path from handling the changes that arrive afterward, so both paths need validation. The Airflow work involved diagnostic runs, investigation with the engineering team, and recorded validation results.

I want the same details available when investigating a model response: the input, the environment, what we expected, and what happened.

I also worked on advertising systems, including Python agent workflows connected to Google Ads operations and backend APIs. An agent response was only one part of the request path. Context, credentials, database access, service boundaries, and returned data all influenced what happened next.

In those systems, checking the answer text alone would miss problems elsewhere in the request. The application might supply incomplete information to the model, or take an inefficient path to produce a reasonable answer. Investigating the response means checking those surrounding steps too.

Those projects inform how I approach testing. LLMCheck is an independent project, and the examples below use synthetic data.

Prerequisites and What We Will Run 🧰

To reproduce this walkthrough, you need Python 3.10 or later, Git and PyYAML. We will use two pinned checkouts: one containing the original experiment scripts, and one containing the 0.3.0 library used in the figures. For a first run with the current tool, use the refund-policy example linked above.

For the offline walkthrough, you do not need an OpenAI API key or the OpenAI Python package. The offline_demo.py script constructs a small object with the same chat-completions shape used by the wrapper. Its response is predetermined. An injected judge then checks predetermined strings.

We’ll start with fixed responses so we can test capture, storage, review and replay without making model requests. 💻 Then we’ll check where the literal judge gets the policy wrong.

Let’s get the example ready. Clone the public source into a fresh directory: 🛠️

git clone https://github.com/nextwebb/llmcheck.git
cd llmcheck
git checkout --detach 986a3118c4c4f6f35d995bb89a8479e32fef0925
git worktree add --detach ../llmcheck-article-0.3.0 6d101ae90781b8dc06965f57313445f8878cf6d6
python3 -m venv .venv
source .venv/bin/activate
python -m pip install 'PyYAML>=6.0'

LLMCHECK_SOURCE="$PWD/../llmcheck-article-0.3.0"
LLMCHECK_EXAMPLES="$PWD/examples/article"
LLMCHECK_OUTPUT="$(mktemp -d)"

The demo script’s --repo argument adds the checkout's src directory to Python's import path, so an editable package installation is not needed for that route.

To use the historical CLI and dashboard shown later, install that worktree in this environment:

python -m pip install -e "$LLMCHECK_SOURCE"
llmcheck --help

The archived example directory contains three main scripts: offline_demo.py runs the example, smoke_test.py checks it, and inspect_results.py prints the saved results. Its evidence directory contains the twelve-answer experiment and confidence checks. These scripts call the LLMCheck library.

Run the following commands in the same terminal so the path variables remain available. LLMCHECK_OUTPUT is a fresh temporary directory; each new run needs an empty directory because the demo refuses to overwrite earlier results.

If you already know the implementation, skip ahead to the twelve-case experiment, confidence, and draft-polarity sections for the checker’s failure cases. 📌

1. Define a Failure Before Writing a Detector 📝

Our teaching policy has two required facts: refunds above $100 need manager approval, and processing takes three to five business days. The question is whether a $150 refund can arrive today. The scripted bad answer is Your refund is instant.

The response has three problems: it omits approval, omits the processing window, and promises an immediate refund. A policy this small lets us check each problem directly.

Write those three requirements into the case. Instructions such as “be accurate” or “do not hallucinate” leave the reviewer to work out what counts as a correct answer each time.

Searching for instant can help you select runs for inspection, but it will also flag “The refund is not instant.” The reviewer or judge still needs to decide whether the response violates the policy.

The demo script flags this scripted failure directly. Later, we try substring matching as the judge and see which answers it gets wrong.

Let’s keep the first case small enough to inspect by hand: 📝

Question: Can I get a $150 refund today?
Policy: Approval required; processing takes 3-5 business days.
Observed response: Your refund is instant.
Failure: Missing required facts and an unsupported timing promise.

Save the question and policy with the response so the next person reviewing it can check the failure against the same requirements.

2. Capture the Response Without Hiding the Test Double 📥

The following excerpt shows the shape of the offline client. The full runnable version is in examples/article/offline_demo.py.

from types import SimpleNamespace

class SyntheticCompletions:
    def create(self, **kwargs):
        return SimpleNamespace(
            choices=[SimpleNamespace(
                message=SimpleNamespace(
                    content="Your refund is instant."
                )
            )]
        )

client = SimpleNamespace(
    chat=SimpleNamespace(completions=SyntheticCompletions())
)

We will wrap this local client with LLMCheck’s instrumentation. This test double returns no token-usage metadata. The recorded latency measures its local function call.

Attach the wrapper and the context you want stored with the call:

from pathlib import Path
from llmcheck import instrument_openai, add_context, add_tags, flag

client = instrument_openai(
    client,
    storage_path=Path("demo-output/fixture.db"),
)

add_context([{
    "id": "synthetic-policy",
    "text": "Refunds above $100 require manager approval "
            "and take 3-5 business days.",
}])
add_tags({
    "workflow": "SYNTHETIC-POC-DEMO",
    "source": "scripted provider; no live model",
})

add_context saves material alongside the next captured call. To give that policy to the model, your application must also include it in the request. Finding a policy in the captured context does not tell you whether the application sent it.

The demo script calls the client and flags the result. The wrapper saves the messages, user input, output, context, tags and available metadata in SQLite after the call succeeds. It captures completion calls, not the rest of the application.

LLMCheck’s optional approved-knowledge feature can prepend system messages before a call. Our fresh demo workspace has no approved knowledge entries. If you enable that feature in an existing workspace, check the stored entries when investigating changes to a request.

3. Run the Offline Demo and Inspect the Results 🧪

Run the demo using the paths set above:

python "$LLMCHECK_EXAMPLES/offline_demo.py" \
  --repo "$LLMCHECK_SOURCE" \
  --output "$LLMCHECK_OUTPUT"

The command generates the captured record, draft, reviewed case, suite and results using LLMCheck’s wrapper, storage, generator, runner and parser. The script performs the review edit so each run produces the same case.

Start with three files:

Inspect the saved CLI display and JSON:

cat "$LLMCHECK_OUTPUT/capture-cli.txt"
python -m json.tool "$LLMCHECK_OUTPUT/captured-run.json"

Look for four things. The output should be the scripted instant-refund sentence. The context should contain the synthetic policy. The tags should identify the run as a demonstration. The flag reason should explain what was wrong, rather than merely say “bad output.”

provenance.json records the source commit and execution mode. IDs and timestamps can change between runs, but the four verdicts should stay the same because the responses and injected judge are fixed.

Run the separate smoke check too:

python "$LLMCHECK_EXAMPLES/smoke_test.py" \
  --repo "$LLMCHECK_SOURCE"

It creates an isolated temporary run and checks capture, review artifacts, expected outcomes, and overwrite refusal. The tests do not open your normal application database. The demo also blocks the Python TCP connection entry points it uses as an accidental-network-call guard. That guard is not an operating-system sandbox for arbitrary untrusted code.

4. Review the Generated YAML as a Specification 📋

Before we approve the generated YAML, let’s read it as a test specification. 📋 Does it describe what a correct future answer must say? A correction written for one bad answer can make a poor test for other answers.

In this revision, the generator places the correction text into must_include and extracts double-quoted phrases into must_not_claim. Review both lists: this rule can misinterpret your feedback, as we will see in the draft-polarity example.

The demo’s scripted review replaces the broad correction with separate requirements:

expected:
  must_include:
    - manager approval
    - 3-5 business days
  must_not_claim:
    - instant
judge:
  type: rubric
  pass_if: >-
    Require manager approval and 3-5 business days;
    do not claim "instant".

This is an excerpt from the reviewed case, not a complete suite file. The demo script also saves identifiers, inputs, context, and metadata. Keeping the generated draft and reviewed version separately lets you see what the script changed.

The offline judge searches for these exact substrings. The default model judge receives the same fields in its prompt, together with the context and rubric, and interprets their meaning.

When reviewing a case, allow correct ways to express the policy. A correct paraphrase may omit the exact phrase manager approval while still saying that a manager must approve the refund. Conversely, a sentence can include both required phrases and negate the policy. We will test both cases shortly.

In a separately configured application workspace, this command reviews the latest capture using that workspace’s llmcheck.yaml and llmcheck_suite.yaml:

llmcheck review --latest

The CLI offers approval, editing, or rejection. Saving valid YAML through the Edit option also approves the case and appends it to the suite; there is no second confirmation. The demo script has already performed its review in code, so you do not need this command to complete the walkthrough.

If an output starts failing, compare it with the policy before editing the case. You may need to fix the application, update the case for a policy change, or correct an expectation that was wrong from the start.

5. Understand What Replay Actually Executes 🔄

The suite identifies a Python function using a module-and-function path:

runner:
  type: python
  callable: fixture_app:answer_user

Here’s the small test application we’ll use to follow the replay: 🔄

OUTPUT = ""

def answer_user(query):
    return OUTPUT

For each scenario, the demo script changes OUTPUT, then invokes the real suite runner. The runner imports the configured function and calls it with the case's inputs as keyword arguments. It evaluates the returned output rather than regrading the archived capture.

In a real integration, that function would call your current application. It could run retrieval, construct messages, and invoke a model. Those operations are your runner's responsibility. The saved case does not automatically restore an old index, database snapshot, tool response, or complete conversation.

To connect your application, expose its answer through an adapter with the same keyword parameter:

# Illustrative adapter: replace this import with your application's API.
from my_app import generate_answer

def answer_user(query):
    return generate_answer(query)

Save it as application_adapter.py beside the suite and make your application package importable in that environment. Change the suite callable to application_adapter:answer_user. The saved case supplies query, so that parameter name must match. This example assumes a synchronous function returning answer text. The my_app import is a placeholder, and this integration pattern was not executed in the offline experiment.

Call your adapter directly with the case's question before running the suite. Inspect its returned text and verify that retrieval and external actions use a controlled test environment. A callable that works in your web server may depend on request state or credentials that the suite process does not have.

Replay reruns the function with the saved inputs. If its answer depends on a catalogue or policy version, your runner must load the version you intend to test.

The scripted passing response produces empty violation lists. The runner uses those lists to return a pass.

The demo script runs four scenarios to check that responses reach the judge and produce the expected verdicts:

You can inspect those decisions without opening the full JSON. Use inspect_results.py to print the saved verdicts:

python "$LLMCHECK_EXAMPLES/inspect_results.py" \
  "$LLMCHECK_OUTPUT/replay-results.json"

The demo run produces the following verdicts:

FAIL | Incorrect scripted response
  Missing: manager approval; 3-5 business days
  Forbidden: instant
  Reason: Synthetic fixture judge: literal matching only; no semantic grounding assessment.
PASS | Corrected scripted response
  Reason: Synthetic fixture judge: literal matching only; no semantic grounding assessment.
FAIL | Missing timing requirement
  Missing: 3-5 business days
  Reason: Synthetic fixture judge: literal matching only; no semantic grounding assessment.
FAIL | Contradictory promise
  Forbidden: instant
  Reason: Synthetic fixture judge: literal matching only; no semantic grounding assessment.
Stored synthetic results only; no application or judge was executed.

The reason is fixed across these scenarios; the missing and forbidden lists show why each case failed. This inspection command reads saved evidence; rerun offline_demo.py with a new output directory to execute the workflow again.

All four results matched the expected verdicts for these scripted responses. This checks the capture-to-replay path; the twelve-case experiment below tests where the literal judge breaks down.

6. Trace the Boundary Between Judge and Verdict 🔎

The default judge is asked to return a JSON object with three violation lists, a reason, and a confidence category:

{
  "missing_requirements": [],
  "forbidden_claims_found": [],
  "unsupported_claims": [],
  "reason": "The required facts are present.",
  "confidence": "high"
}

That object is an illustrative payload shape. In the offline demo, a test function supplies the JSON. In the default path, an OpenAI model supplies it. Both paths use the same response parser.

The parser requires lists of strings, a string explanation, and a confidence value of high, medium, or low. It rejects responses with missing fields or incorrect types.

The aggregation rule then asks whether any violation list is nonempty:

violations = (
    judge_result.missing_requirements
    or judge_result.forbidden_claims_found
    or judge_result.unsupported_claims
)
passed = not bool(violations)

A reported violation fails the case. Empty lists pass it. The suite passes when all its cases pass.

Here’s the catch. 🤔 If the judge overlooks a contradiction and returns empty lists, this rule still passes the case. The Python code does not check the policy again.

7. Case Study: Twelve Answers That Challenge the Demo Judge 🧩

The first four cases checked the workflow. I then used twelve cases to test how the literal judge handled paraphrases, negation, contradictions and extra promises. Each case had a policy label assigned before the run.

The policy for this experiment also rules out guaranteed approval and a promised cash bonus. The judge still checks two required substrings and one forbidden substring, and always returns an empty unsupported-claims list.

For now, we’re testing the injected literal judge. We’ll try the default model judge in section 12. 🧪

The categories were literal compliant answers, compliant paraphrases, compliant negations, missing required facts, contradictions containing the expected words, and unsupported extra promises. There were two cases in each category.

Four decisions agreed with the predefined labels and eight disagreed across these twelve selected cases.

Consider a paraphrase: “For a refund over $100, a manager must approve it, and processing takes three to five business days.” The policy meaning is preserved. The literal checker fails it because neither required phrase appears in the exact expected form.

Now consider negation: “Refunds above $100 require manager approval and take 3-5 business days. They are not instant.” The forbidden substring is present, so the checker fails a response that explicitly denies the prohibited promise.

A contradiction gets through: “Manager approval is not required for refunds above $100; processing takes 3-5 business days.” Both required strings appear, and the forbidden one does not. The checker passes a response that reverses the approval policy.

The judge also passes responses that state the required facts and then guarantee approval or promise a $25 cash bonus. Its unsupported-claims list is always empty, so neither promise affects the verdict.

The result is four false passes, four false failures, two correct passes, and two correct failures. The labels come from the refund policy defined for this experiment.

8. Case Study: A Low-Confidence Result Still Passes 🤔

What happens when the judge isn’t confident? 🤔 It returns a confidence field, but this revision does not use it to decide pass or fail.

The next probe passed four injected judge responses through the actual suite runner and parser. High confidence with empty violation arrays passed. Low confidence with empty arrays also passed. A reported violation failed. Malformed JSON produced an execution error.

In the reported-violation probe, the injected judge claimed a required phrase was missing even though the application output contained it. The case failed because the verdict logic uses the judge's violation lists without independently checking the answer.

The left side shows the probe results. The right side shows a proposed review state, which is not implemented.

I would add a review state alongside pass, fail and error. That would let a low-confidence response wait for review even when its violation lists are empty.

The values high, medium, and low are labels, not measured probabilities. We have not checked whether this judge's high-confidence verdicts are more often correct than its low-confidence ones, so the labels alone cannot tell us how dependable that review rule would be.

9. Case Study: A Draft That Contradicts Its Correction ✏️

The 0.3.0 generator produced a contradictory draft from this correction. The merged implementation fixes the defect, but the old output shows why draft review matters:

Include "manager approval".

Now look at the generated criteria. 🔍 They included the full correction as a required entry and extracted the quoted phrase as forbidden:

expected:
  must_include:
    - Include "manager approval".
  must_not_claim:
    - manager approval

The draft requires manager approval while also forbidding it. The 0.3.0 extraction code treated double-quoted text as forbidden without checking whether the correction asked to include it.

The synthetic record reproduced the 0.3.0 defect. The repaired criteria illustrate a manual edit; the merged code fix works differently, as described below.

Both lists are valid YAML, so the parser accepts them. A reviewer could repair this case by requiring manager approval, clearing the forbidden list, and checking that the rubric agrees with those changes.

The merged fix stops extracting forbidden claims from quotation marks. It keeps the correction in the rubric and leaves both phrase lists empty for the reviewer.

10. Keep the Application, Storage and Provider Boundaries Visible 🔐

LLMCheck stores records locally. Its default judge sends evaluation material to OpenAI, and your application runner may make its own external calls.

The offline demo supplies fixed responses. The normal judge path crosses the provider boundary with inputs, context, output, criteria and rubric.

For the refund case, the judge needs the policy and the answer. It does not need a customer credential or an unrelated internal note. Choose what to capture with the later evaluation request in mind.

Keep a stable identifier and policy version with the context, along with the application and judge configuration. If the approval threshold changes from $100, those records let you check whether a failure comes from the application or an outdated case.

Replay executes your Python function. If that function sends email, changes a campaign, or writes to production, replay can repeat those actions. Use a sandboxed application path or substitute those actions during development evaluations.

The suite names the module and function to execute, and the runner makes the suite directory available during import. Inspect that code before running a suite from an unfamiliar source.

11. Regression Cases and Reusable Knowledge Are Different Artifacts 📚

The repository also has an experimental workflow that turns reviewed information into knowledge entries for later requests.

A regression case replays an input and checks the answer against reviewed expectations. An approved knowledge entry can supply information to later application requests.

The pilot classifies failures by cause: missing context, wrong retrieval targets, reasoning failure despite sufficient context, prompt or instruction failures, workflow failures, ambiguity, and false-positive detectors.

An answer that contradicts a supplied refund policy may not improve when given another copy of that policy. Retrieving the wrong document calls for checking retrieval constraints. An ambiguous policy needs clarification before it becomes reusable guidance.

At this revision, matching uses workflow scope, usage mode, and token overlap rather than semantic search. Certain approved modes prepend planning guidance or runtime context as system messages before the original application messages. The run metadata records which entries were applied. Detection-only and review-assist modes do not inject knowledge.

After installing the historical CLI above, open the dashboard against the generated workspace:

llmcheck pilot-dashboard \
  -c "$LLMCHECK_OUTPUT/llmcheck.yaml" \
  --host 127.0.0.1 --port 8765

The dashboard from an earlier isolated run of the same synthetic demonstration at the pinned revision. Its run ID and timestamp differ from a new demo run. The review count and percentages come from fixture data.

The demo script supplies one synthetic review to inspect in the dashboard. I have not measured whether knowledge reuse reduces repeated failures. That would need repeated cases, a baseline without the reused knowledge, and a comparison of both corrected answers and new errors.

12. Move to a Live Judge as a Separate Experiment 🌐

On 23 September 2026, we ran a separate evaluation using the default OpenAI judge. The script sent the twelve refund-policy cases to gpt-4o-mini-2024-07-18, once each, with no retries. All twelve requests completed.

The recorded outcomes were five correct passes, six correct failures, zero false passes and one false failure. The rejected compliant response included the sentence “Do not expect an instant refund.” The judge reported the negated phrase as forbidden. The saved run includes that false failure.

The public explorer shows each saved response in its Recorded OpenAI run mode. Clicking through those results does not repeat the evaluation, and the server needs no provider key. These selected examples show how this configuration handled the refund policy; they are too narrow to estimate accuracy on other tasks.

To use the live judge with your application, both 0.3.0 and the merged implementation require an OpenAI API key with access to the configured judge model. This is separate from the offline walkthrough. The environment above has the 0.3.0 CLI installed; these commands do not upgrade it. Follow the repository installation instructions in a separate environment to use the current implementation.

Set OPENAI_API_KEY in the environment used to run the suite. Keep the key out of case files and source code. Set judge.model in llmcheck.yaml to choose the model; the configuration defaults to gpt-4o-mini when that field is omitted.

Before using the CLI, prepare a configuration with a real judge.model and a suite pointing to your application adapter from section 5. The offline demo uses unused-in-fixture-demo as its model name, and its saved fixture function starts with an empty response. Adding an API key does not turn those files into a live application test.

With your configuration saved as llmcheck.yaml and your suite as llmcheck_suite.yaml, run:

llmcheck run-suite -c llmcheck.yaml --suite llmcheck_suite.yaml

This command calls OpenAI through the default judge, even when your application function returns a fixed string. The offline demo supplies its own judge through the Python API; the CLI has no equivalent offline switch.

For a follow-up, I would add cases that were not used to adjust the prompt, covering paraphrases, negation, unsupported additions and contradictions. The twelve cases and their authored labels stayed fixed during this run.

Save the judge model, prompt version, case-set version, source commit, full payload and returned JSON so you can compare runs. Repeating requests under the same configuration would show whether the verdicts vary. The source uses temperature zero, which does not guarantee identical remote responses.

Keep false passes and false failures separate. In this run, the one disagreement rejected a compliant answer. A false pass on “manager approval is not required” would have a different consequence: accepting an answer that reverses the policy.

Latency, token usage and cost were not measured in this run. The judge transport returns response content without token-usage data. To compare costs or timings, instrument those values separately and use the same case set for each configuration.

13. Make the CI Contract Explicit ⚙️

The CLI returns zero when all cases pass, one when at least one case fails, and two for caught configuration or suite-execution errors. Code two covers handled import, runner and judge failures; uncaught exceptions are not universally mapped to it.

A case that fails because the answer promises an instant refund needs an application or expectation review. A missing module or malformed judge response needs an execution fix. Both may block a release, but the saved error should tell you which problem to investigate.

Parser and aggregation tests use controlled inputs, so they can run on every change without a provider call. Choose when to run the live judge based on its cost and how often the application or evaluation criteria change.

Review cases before adding them to a release gate. “Refunds are instant” and “Do not expect an instant refund” contain the same forbidden word, but test different meanings, so keep both. Repeated copies of the same promise add less coverage. When the policy changes, update the affected expectations and record why.

Save the application output, judge payload, reason and configuration for failures. Those files let you see whether the judge rejected a negation or found an actual promise.

Something didn’t behave as expected? Here’s where to start looking for the failures we covered: 🧰

14. What Passed, What Failed, and What Remains Open 📌

The offline run saved the capture, generated and revised the case, and executed the four replay scenarios. All four verdicts matched their expected results. The separate smoke test also confirmed that a second run cannot overwrite the first run’s output directory.

The pinned 0.3.0 run passed 21 tests with provider fakes or mocks. The merged hardening revision passed 58 tests. Separately, the twelve OpenAI requests agreed with eleven authored labels and rejected one compliant answer containing a negation. The unit tests and model evaluation check different behavior.

Wrapping Up 😊

Start with the complete negation case when you try a different judge. Its answer includes the required approval and timing facts, followed by “Do not expect an instant refund.” The recorded judge rejected that compliant answer. Keep it alongside a response that actually promises an instant refund, and inspect the reasons for both verdicts.

Your turn. 😊 Add a failure from your own application, write down the expected behavior, and check it again after your next change. 🧪

Source: github.com/nextwebb/llmcheck. To try the current tool, run the refund-policy example. It captures a scripted response, saves a reviewed case and prints the replay results.

Have feedback or a case you want to discuss? Leave a comment 💬 or reach me on GitHub, Twitter/X, or LinkedIn. If this helped, give it a thumbs up 👍 and share it with someone testing their own LLM application.

If you have applied to many engineering roles in a short period, you already know the routine.

You find a role that looks right. You read the job description. You adjust your CV. You write a cover letter that should sound specific, not generic. You upload a PDF. Then you type the same name, phone number, location, LinkedIn URL, GitHub URL, work authorization answer, and application note into another ATS form.

Then you do it again.

After a while, the work becomes less about judgment and more about repetition. That is the part I wanted to remove.

I built career-agent to handle the mechanical layer of job applications from inside Claude Code. It helps you build a local profile, source roles, scaffold role configs, generate tailored PDFs, fill supported ATS forms in Chrome, and track your pipeline.

It does not submit applications for you.

That boundary matters.

The plugin helps you prepare the application. You still review the CV, review the cover letter, answer sensitive questions yourself, and click Submit only when you are ready.

In this guide, I will walk through how to install it, what each command does, where the files live, what I tested, and where you should still slow down.

What career-agent Actually Is

career-agent is a Claude Code plugin for job application workflows.

It gives you six slash commands:

/setup-profile
/source
/new-role
/generate-cv
/apply
/track

The plugin is not a SaaS app. It is not a hosted job board. It is not a browser extension by itself.

It runs locally through Claude Code and writes files into your workspace:

profile.json
roles/
generated/
tracker.json

Those files are gitignored by default because they can contain personal data, role notes, generated CVs, cover letters, and application history.

That local file model is intentional. I wanted the workflow to be inspectable. If Claude generates a role config, you can open the JSON file. If the CV does not sound right, you can inspect the profile data and regenerate. If an application status changes, you can see it in tracker.json.

This is the high-level command map:

• /setup-profile builds profile.json from your CV, LinkedIn export, or pasted profile text
• /source helps find and shortlist roles based on your profile and search terms
• /new-role creates a structured roles/<role_id>.json file from a job URL
• /generate-cv generates a tailored CV and cover letter PDF
• /apply fills supported ATS forms in Chrome, then stops for your review
• /track updates the local application pipeline

The supported ATS platforms today are Greenhouse, Lever, and Workable.

That does not mean every page variant will work perfectly. ATS pages differ by company, region, custom fields, embedded forms, and hidden inputs. Treat /apply as a form-filling draft, not as a final application.

The strongest way to think about career-agent is this:

Claude handles the repeated structure.
You handle the judgment.

That means you still decide which roles are worth applying to, what story the cover letter should tell, what sponsorship or location answers are accurate, and when to submit.

Why I Built It

I apply to a lot of roles.

When you apply at that volume, the problem is not writing one good CV. The problem is maintaining quality while repeating the same application mechanics many times.

The work breaks into two parts:

1. Judgment work: choosing roles, deciding what experience to highlight, checking whether the company is a fit, and making sure the final application is honest.
2. Mechanical work: copying fields, generating PDFs, formatting cover letters, creating folders, tracking status, and filling the same forms again.

Most of the stress comes from the mechanical work leaking into the judgment work.

You start with good intent, but after the twentieth application, small mistakes become easier. Maybe the summary still mentions the previous company. Maybe the cover letter has a generic line. Maybe the tracker is missing a status. Maybe you cannot remember whether you already applied to that role.

career-agent is my attempt to separate those layers.

I want Claude to help with the repeatable workflow, but I do not want Claude to own the irreversible action.

That is why the tool stops before Submit.

What Was Actually Tested

For this article, I used a mix of real local runs and documented command behavior.

The install flow was run in a disposable directory with:

npx @nextwebb/career-agent

That verified Python detection, reportlab, Claude Code CLI detection, and profile.json creation.

The generation and tracker screenshots were also produced from a clean temporary workspace using example profile and role data. That means the screenshots do not expose real personal data.

The article screenshots show:

• A real npx @nextwebb/career-agent install run
• A role config list from a clean example workspace
• A real PDF generation command using the example role
• A tracker command adding and marking an example role as applied

What I did not do for this article:

• I did not run /apply against a live company application form while writing this article
• I did not submit a real job application for the tutorial
• I did not use real personal profile data in the screenshots

That distinction matters because a tutorial should not pretend that a browser automation flow is safer or more universal than it is. The local generation and tracking flows are easy to verify. Live ATS automation depends on the page in front of you.

Prerequisites

Before installing career-agent, ensure you have these tools installed:

Node.js 18+
Python 3.10+
Claude Code CLI
Claude in Chrome extension, required for /apply

Check your local versions:

node --version
python3 --version
claude --version

Node is required because the setup entry point runs through npx.

Python is required because the PDF generation uses reportlab.

Claude Code is required because this is a Claude Code plugin. The slash commands live inside Claude Code, not inside a standalone web app.

Claude in Chrome is required only for /apply. You can set up your profile, create role configs, generate PDFs, and track applications without touching Chrome. But if you want Claude to fill ATS forms, the Chrome extension needs to be connected to Claude Code.

Note that being signed in to Claude.ai is not enough. Check the Claude in Chrome extension connection before depending on /apply. 📌

1. Install career-agent

Create a workspace for your job application workflow:

mkdir ~/career-agent-workspace
cd ~/career-agent-workspace

Then run the npm setup command:

npx @nextwebb/career-agent

npm package: @nextwebb/career-agent 📦

The setup script checks your local environment and creates a starter profile.json in the current directory.

Specifically, bin/setup.js checks:

• Python 3.10 or newer
• reportlab
• Claude Code CLI on your PATH
• Whether profile.json already exists

If reportlab is missing, the setup tries to install it with pip. If Claude Code is missing, it stops and tells you to install it first.

This matters because the plugin is not useful if the PDF generator cannot run or if Claude Code cannot see the plugin commands.

Here is the same install flow from a disposable terminal run:

After the npm setup, register the plugin source:

claude plugin marketplace add nextwebb/career-agent

That command registers the GitHub repo as a Claude Code plugin source. I am not presenting career-agent as an official Anthropic marketplace listing. The evidence-backed claim here is narrower: the installer and manual command use the GitHub-hosted plugin source.

Then install the plugin:

claude plugin install career-agent

Restart Claude Code after installing.

Open a new Claude Code session, type /, and check that the commands appear.

You should see:

/setup-profile
/source
/new-role
/generate-cv
/apply
/track

If they do not appear, check Claude Code plugin settings and confirm that the plugin source was added correctly. ✅

2. Set Up Your Profile

The profile is the center of the workflow.

Run:

/setup-profile

Claude will ask for your CV, LinkedIn export, or pasted profile text. It then creates profile.json.

The profile file is more than a contact card. It stores your name, location, links, experience, education, skills, summaries, and CV variants.

The goal is to give Claude a structured source of truth, so every role-specific output starts from the same base.

A simplified profile looks like this:

{
  "name": {
    "first": "Jane",
    "last": "Doe"
  },
  "email": "jane@example.com",
  "location": "Berlin, Germany",
  "headline": "Senior Backend Engineer",
  "summary": "Backend engineer with experience building APIs and distributed systems.",
  "links": {
    "github": "https://github.com/janedoe",
    "linkedin": "https://www.linkedin.com/in/janedoe"
  },
  "experience": [],
  "education": [],
  "skills": []
}

The actual schema is larger because it supports variants and per-role bullet sets.

career-agent uses three audience variants:

• A: AI, LLM, and evaluation roles
• B: data platform and pipeline roles
• C: senior backend and API roles

That is useful because the same experience can be framed differently depending on the role.

For example, a platform migration can matter to backend teams because of reliability and APIs. The same migration can matter to data teams because of pipeline stability and data quality. The facts stay the same, but the emphasis changes.

Review profile.json after generation.

Do not skip this.

PDF extraction is useful, but it is not magic. Dates can be misread. Role titles can be merged. Links can be missed. A summary can sound too broad. You should fix titles, dates, links, and anything that does not sound like you.

This is also the right time to remove anything you do not want used in generated applications.

3. Understand the Local Files

Before running more commands, it helps to understand what the workspace will look like.

The important paths are:

profile.json
roles/
generated/
tracker.json

profile.json is your base profile.

roles/ contains one JSON file per role.

generated/ contains the CV and cover letter PDFs.

tracker.json stores application status and notes.

All of these are local files. They are also gitignored by default.

This design makes the workflow easy to inspect. If /generate-cv produces a weak PDF, you can inspect the profile and role config. If /apply fills a field badly, you can inspect custom_answers in the role config. If your pipeline feels messy, you can open tracker.json.

That is one of the main reasons I prefer this setup over a hidden workflow.

4. Source Roles with /source

If you already have a job URL, you can skip /source and start with /new-role.

If you want career-agent to help build a shortlist, run:

/source Germany backend

You can change the location and role type:

/source remote data platform
/source UK AI infrastructure
/source Netherlands backend

The sourcing command uses your profile as the baseline. It is meant to help find roles that match your background, not every role on the internet.

The expected output is a ranked list of roles with direct application URLs, fit notes, and signals around location, sponsorship, and role match.

Use this output as a starting point.

Still verify the important details yourself:

• Is the role currently open?
• Is the application URL direct?
• Does the location work for you?
• Does the role mention sponsorship or relocation?
• Is the role actually senior enough?
• Does the company match what you want?

I do not treat sourcing output as final truth. I treat it as a shortlist.

That distinction matters because job pages change, roles close, and sponsorship language is often vague.

5. Create a Role Config with /new-role

For a role you want to apply to, run:

/new-role <job_url>

This creates a role config under roles/.

The config includes:

• Role ID
• Company name
• Title
• Location
• Job URL
• ATS platform
• CV variant
• Output prefix
• Cover letter stubs
• Custom answer fields

A role config gives the rest of the workflow a stable reference. Instead of asking Claude to remember a job description across a long conversation, the role details live in a file.

A simplified role config looks like this:

{
  "role_id": "stripe_backend_2026",
  "company": "Stripe",
  "title": "Senior Backend Engineer",
  "location": "Remote, Europe",
  "url": "https://boards.greenhouse.io/example/jobs/123",
  "ats_platform": "greenhouse",
  "variant": "C",
  "custom_answers": {
    "hear_about_us": "LinkedIn",
    "visa_sponsorship": "",
    "work_authorization": "",
    "why_company": ""
  },
  "cover_letter": {
    "paragraphs": []
  }
}

This is where I slow down.

The scaffold is a starting point. It is not the final application.

Edit the cover letter stubs. Fill in why_company. Add accurate work authorization details. Add salary expectations only if you are comfortable doing so. If the job post includes a reading-check instruction, make sure it is represented correctly.

The screenshot below uses the example role config in a clean local workspace:

6. Generate the CV and Cover Letter

Once profile.json and roles/<role_id>.json are ready, run:

/generate-cv <role_id>

Under the hood, the generator reads your profile and the role config, then writes two PDFs into generated/.

The output format is:

generated/<output_prefix>_CV.pdf
generated/<output_prefix>_CoverLetter.pdf

The CV is intentionally ATS-safe:

• Single column
• No tables
• No images
• No decorative layout tricks
• Clickable links where relevant
• Built with reportlab

That plain layout is not an accident.

Many ATS parsers still struggle with multi-column CVs, text boxes, icons, and complex PDF structure. A clean single-column PDF is less exciting visually, but it is easier for machines and recruiters to parse.

Here is the terminal output from generating the example CV and cover letter:

Open both PDFs before applying.

Check:

• The summary
• The role title
• The ordering of experience
• The skill list
• The dates
• The links
• The cover letter tone
• Any claims that sound stronger than your actual experience

The goal is not to invent experience.

The goal is to present the most relevant parts of your real experience for that role.

If the PDF sounds too generic, go back to the role config. If it sounds inaccurate, go back to profile.json. The files are local because you are supposed to inspect and edit them.

7. Fill the ATS Form with /apply

For supported ATS pages, open the application in Chrome and run:

/apply <role_id>

career-agent reads the form, maps fields from your profile and role config, and fills what it can.

The supported platforms today are:

Greenhouse
Lever
Workable

The important boundary is this:

Claude fills fields.
You review every field.
You answer EEO and voluntary questions.
You click Submit.

That boundary is deliberate.

Applications often include questions about gender, race, veteran status, disability status, sponsorship, work authorization, consent, relocation, and salary. Some of those are sensitive. Some of them have legal implications. Some of them need context that no tool should guess.

Claude should not answer those for you.

Claude should also not click Submit.

When /apply works well, it saves you from retyping the boring parts. It fills names, links, locations, uploads, and text fields that map cleanly to your local config.

But you still review.

Treat the filled page as a draft. ATS forms vary, even on the same platform. One company may use standard Greenhouse fields. Another may add custom dropdowns, hidden file inputs, or React comboboxes. Review everything before submitting. ❌

If a page is not Greenhouse, Lever, or Workable, you can still use the generated PDFs and tracker manually.

8. Track the Application

After submitting, update the tracker:

/track update <role_id> status=applied notes="Applied today"

You can also run:

/track

tracker.json gives you one local place for statuses, notes, dates, and follow-ups.

That becomes useful quickly.

When you are applying to many roles, memory is not enough. You need to know which roles are drafts, which ones you applied to, which ones have phone screens, which ones are rejected, and which ones need follow-up.

The tracker supports statuses like:

draft
applied
screen
interview
offer
rejected
withdrawn

Here is a local tracker run using the example role:

The tracker is not meant to replace a full CRM. It is meant to keep the application workflow from becoming scattered across memory, browser tabs, and random notes.

How the Full Workflow Connects

Here is the full flow:

1. Build profile.json with /setup-profile
2. Find or capture a role with /source
3. Create roles/<id>.json with /new-role
4. Generate PDFs with /generate-cv
5. Fill the form draft with /apply
6. Save status with /track

The value is not only one command.

The value is the chain.

/setup-profile gives you a structured base.

/source helps you find roles.

/new-role turns a job URL into a local config.

/generate-cv turns the profile and role config into PDFs.

/apply fills the supported ATS form draft.

/track keeps the status visible.

Because each stage writes or reads a local file, you can stop at any point and inspect the work.

That is important for trust.

If a tool hides the intermediate steps, you have to trust the whole output. If the tool gives you files, you can inspect the profile, inspect the role config, inspect the PDFs, and inspect the tracker.

That is the workflow I wanted.

A Practical Example Flow

Let us say I want to apply for a senior backend role in Europe.

I start with sourcing:

/source Germany backend

From that output, I pick a role that looks relevant.

Then I scaffold it:

/new-role https://boards.greenhouse.io/example/jobs/123456

I open the generated role config:

roles/example_backend_2026.json

I check the company name, role title, location, ATS platform, and custom answer fields.

Then I edit the cover letter paragraphs. I want them to reference something real: the company product, the infrastructure problem, the role requirements, or a public engineering post.

After that, I generate the PDFs:

/generate-cv example_backend_2026

I open the CV and cover letter.

If the summary sounds too generic, I edit the role config. If the experience bullets feel wrong, I update the profile. If the cover letter feels like filler, I rewrite it.

Only after that do I open the application page and run:

/apply example_backend_2026

Claude fills what it can.

I review the form, answer EEO fields myself, check the uploaded files, and submit manually.

Then I track it:

/track update example_backend_2026 status=applied notes="Applied after reviewing form"

That is the loop.

It is not magic. It is a structured local workflow.

What I Like About This Approach

The first thing I like is that the data stays local.

Job application data can be personal. It includes phone numbers, locations, employment history, immigration details, compensation expectations, and private notes. Keeping that in local gitignored files makes the workflow easier to reason about.

The second thing I like is that the tool has a natural review point.

The generated PDFs are not hidden. You open them. You read them. You decide whether they represent you well.

The third thing I like is that /apply has a human boundary.

I do not want a tool clicking Submit on my behalf. It is too easy to get one field wrong and send a bad application. It is also not appropriate for a tool to guess voluntary self-identification answers.

The fourth thing I like is that the workflow is composable.

You do not need to use every command every time. If you already have a role URL, start with /new-role. If you only want PDFs, stop after /generate-cv. If the ATS is unsupported, fill the form manually and still use /track.

That flexibility matters because real job applications are messy.

How I Review an Application Before Submitting

The review step is the part I do not automate.

That might sound strange after building a tool to automate job application work, but this is the boundary that makes the workflow usable for me.

When /apply fills a form, I treat the result as a draft. I do not treat it as a completed application. That mindset matters because even a good automation can misunderstand a field label, miss a hidden requirement, or map a custom question too broadly.

Here is the review checklist I use before I submit:

1. Confirm the company and role title
2. Confirm the uploaded CV and cover letter are the right files
3. Read every pre-filled text field
4. Check phone, email, location, and links
5. Check work authorization answers
6. Answer EEO or voluntary self-identification fields manually
7. Check custom screening questions
8. Check final consent boxes
9. Submit manually
10. Update tracker.json

The first check sounds obvious, but it catches mistakes.

If you are applying to several roles in one session, it is easy to generate files for one role and open a different tab. The file names help, but I still check the company name and title before moving forward.

The second check is the uploaded file check.

Many ATS forms show the uploaded filename after upload. I check that the CV and cover letter match the current role ID. If I see the wrong role name in the filename, I stop. It is better to catch that before Submit than to explain it later.

The third check is every text field.

I read the filled fields the way a recruiter would see them. Not as code. Not as JSON. As a person reading an application. If the answer feels too long, too vague, or too generic, I edit it in the form before submitting.

Work authorization deserves its own pause.

Do not let a tool guess this. If the role asks whether you need sponsorship now or in the future, answer based on your actual status and the wording of the form. If the wording is unclear, slow down and decide carefully. The tool can carry your configured answer, but it should not decide the answer for you.

EEO and voluntary self-identification fields are also yours to answer.

career-agent is designed not to fill those fields. That is intentional. Those fields can be sensitive, jurisdiction-specific, and personal. If you want to answer them, answer them yourself. If you prefer not to answer, choose that option yourself.

The final thing I check is consent.

Some applications include checkboxes for privacy policies, talent pools, communication consent, or data retention. I do not want automation accepting those for me without review. Read the labels. Decide what you agree to. Then submit.

After submission, I update the tracker immediately:

/track update <role_id> status=applied notes="Submitted after manual review"

That note looks small, but it helps later.

When you are applying at volume, the tracker becomes memory. It tells you what happened, when it happened, and what you need to follow up on.

How I Use career-agent During an Application Week

The best way to use this tool is not to rush from job URL to Submit.

For me, the better workflow is batching.

I separate the week into sourcing, preparation, generation, review, and submission. That keeps the work calm. It also reduces mistakes because I am not trying to research a company, rewrite a cover letter, generate PDFs, fill a form, and update a tracker in one tired session.

Here is a realistic week:

Monday: source roles and build a shortlist
Tuesday: create role configs
Wednesday: edit cover letters and custom answers
Thursday: generate PDFs and review them
Friday: apply, submit manually, and update tracker

You do not need to follow that exact schedule, but the separation is useful.

On sourcing days, I focus on finding roles that are worth the time.

I might run:

/source Germany backend
/source remote AI infrastructure
/source UK data platform

Then I read the shortlist and remove roles that do not fit. I am not trying to apply yet. I am trying to decide what deserves attention.

On role config days, I run /new-role for the strongest matches:

/new-role https://boards.greenhouse.io/example/jobs/123
/new-role https://jobs.lever.co/example/456

That gives me structured files under roles/.

At this point, the job application becomes editable data. I can inspect the title, location, ATS platform, CV variant, and custom answer fields. I can also keep the job description context close to the generated application materials.

On writing days, I edit the role configs.

This is where the application becomes specific.

I do not want a cover letter that says, "I am excited about your mission" and nothing else. I want the letter to mention the actual engineering problem, the product surface, the infrastructure shape, or the team context where I can add value.

For example, instead of this:

I am excited about the opportunity to contribute to your team.

I would rather write this:

Your backend platform work around payments, reliability, and developer-facing APIs maps closely to the systems work I have done across Python, Go, AWS, and Kubernetes.

That second version is still simple. It does not pretend to know private company details. But it is specific enough to show that the application was not generated blindly.

On generation days, I run:

/generate-cv <role_id>

Then I open the generated PDFs.

I do not review PDFs in bulk without reading them. That defeats the point. The generation step saves time, but the review step protects quality.

On submission days, I open the application forms and run /apply.

The goal is not to submit as fast as possible. The goal is to remove repeated typing so I can spend my energy reviewing the parts that matter.

That distinction is important.

If you are applying to ten roles in a day, the worst outcome is not that the work takes longer than expected. The worst outcome is sending ten weak or incorrect applications quickly.

Keeping profile.json Honest

profile.json is the base layer of the whole workflow.

If it is inaccurate, every generated output can inherit that inaccuracy.

That means the profile deserves careful review before you rely on it.

After running /setup-profile, I check these sections first:

name
email
phone
location
links
headline
summary
skills
experience
education
variants

The easy fields are name, email, phone, and links. Still check them.

Phone numbers can lose country codes. LinkedIn URLs can be pasted without the full path. GitHub URLs can be wrong if your CV had an old link. These are small issues, but they are visible to recruiters.

The summary needs more attention.

A generated summary can sound correct but still not sound like you. It might over-index on one part of your background. It might sound too broad. It might use words you would not use in an interview. Edit it until you can read it out loud without feeling awkward.

Skills are another place where honesty matters.

I prefer listing skills I can defend in a technical conversation. If a tool extracts every technology that appears once in a project bullet, the skill list can become noisy. Remove tools you only touched lightly. Keep the ones that represent your real working experience.

Experience bullets deserve the most time.

A good bullet should include what you did, the system or product area, and the outcome where you can support it.

This is weak:

Worked on backend APIs.

This is better:

Built and maintained Python backend APIs for internal workflow automation, reducing repeated manual operations for support and operations teams.

If you have numbers, use them. If you do not have numbers, do not invent them.

It is better to write a clear factual bullet than to add a fake metric.

The variant sections are where career-agent becomes more useful.

Variant A can emphasize AI and evaluation work. Variant B can emphasize data platforms and pipelines. Variant C can emphasize backend systems and APIs. The facts should stay the same, but the ordering and framing can change.

That is normal.

Different roles care about different parts of the same background. The important part is that every variant remains true.

I also recommend keeping a clean backup of a reviewed profile:

cp profile.json profile.reviewed.json

Note that both files can contain personal data, so keep them in the gitignored workspace. Do not commit them.

Before a serious application batch, I run a quick profile check:

python3 -m json.tool profile.json > /tmp/profile.checked.json

That verifies the JSON syntax. It does not verify the content, but it catches broken commas, missing quotes, or accidental edits that would break the workflow.

How to Think About /source

/source is useful, but I do not treat it as a replacement for judgment.

I treat it as a shortlist builder.

That distinction keeps expectations realistic.

A sourcing command can search based on your profile, location, and keywords. It can return roles that look aligned. It can help organize signals like location, sponsorship wording, company, title, and fit score.

But job markets are messy.

Roles close without notice. Companies repost old roles. Some career pages keep expired roles indexed. Some job boards cache listings after the company page has changed. Some postings mention remote work in one section and office requirements in another.

So when /source returns a role, I still verify:

Is this role still live on the company site?
Is the application URL direct?
Is the location compatible with me?
Does the role match my level?
Does the role match the kind of work I want?
Is sponsorship or relocation actually stated?

I am careful with sponsorship language.

If a job post explicitly says sponsorship is available, that is a strong signal. If an uploaded company list says a company has sponsored before, that is not the same thing. It can help you decide where to look, but it does not prove this role sponsors now.

That is why the role config still needs review.

Use /source to reduce random browsing. Use your judgment to decide whether the role deserves a tailored application.

Here is the flow I recommend:

/source <location> <role_type>
read the shortlist
open the strongest company pages
verify the role is live
run /new-role on the real application URL
edit the role config
generate PDFs
apply with review

By using /source this way, you keep the benefit without pretending it knows everything.

Reviewing Generated PDFs Like a Recruiter

Generated PDFs should be reviewed visually and textually.

First, I read the CV as a person.

I ask:

Can I understand the role target in the first 10 seconds?
Does the summary match the job?
Are the strongest experiences near the top?
Do the skills match the role without looking stuffed?
Are the bullets specific?
Are dates and titles correct?
Are links present and readable?

Then I review it as an ATS would parse it.

That means I avoid being impressed by visual decoration. The layout is supposed to be plain. A single-column CV might look less fancy than a design-heavy resume, but it is easier to parse.

If you want to test extraction, you can use a local PDF text tool:

pdftotext generated/<file>.pdf - | head -80

If pdftotext is not installed, skip this check or install Poppler through your package manager. This is optional, but useful when you want confidence that the PDF text is extractable.

For the cover letter, I check tone.

A cover letter should sound like a human wrote it for this role. It does not need to be dramatic. It needs to be specific, honest, and easy to read.

I look for these problems:

• It repeats the CV without adding context
• It sounds too generic
• It claims excitement without evidence
• It mentions technology the company does not appear to use
• It overstates my experience
• It has placeholder text
• It is too long for the role

If I find those issues, I do not edit the PDF directly. I edit the role config or profile, then regenerate.

That keeps the source of truth clean.

Using the Docs Page Alongside the Article

This article is meant to teach the workflow from a practical angle.

The docs page is there for quick reference:

nextwebb.github.io/career-agent 📌

I would use both together.

Use the article when you want the full flow and the reasoning behind the boundaries. Use the docs page when you want the command list, repository context, and landing page reference.

That is why I wanted the GitHub Pages site live before publishing this.

If someone reads the article and wants to check the project quickly, they should have somewhere clear to go. The repo is the source of truth for code. The npm page is the source of truth for the package. The docs page gives the project a cleaner entry point.

The three links serve different jobs:

GitHub repo: code, issues, PRs, license
npm package: installable package metadata
docs page: quick project overview

Here they are:

github.com/nextwebb/career-agent

@nextwebb/career-agent 📦

nextwebb.github.io/career-agent 📌

How Contributors Can Help

The most useful contributions are not cosmetic.

The best contributions improve one of three things:

ATS coverage
field mapping reliability
documentation from real usage

ATS coverage matters because companies use different systems.

Greenhouse, Lever, and Workable are handled today. But many companies use Ashby, iCIMS, SmartRecruiters, Taleo, BambooHR, custom Workday flows, or embedded forms. Each platform has its own field patterns, file upload behavior, and validation rules.

If you want to contribute a new ATS adapter, start by documenting the page shape.

Useful notes include:

ATS platform name
example public application URL
field labels shown on page
input names or IDs if visible
file upload behavior
required fields
custom question behavior
submit button behavior
fields the tool should never fill

Do not include private applicant data in an issue.

If the page can be shared publicly, include the job URL. If it cannot, describe the structure and redact sensitive information.

Field mapping reliability is the second big area.

This is where small improvements matter. A field labeled "Preferred name" should not always map to legal first name. A field labeled "Current company" should not map to target company. A field labeled "Website" might accept GitHub, personal site, or portfolio depending on the form.

These are not glamorous problems, but they are the real problems in application automation.

Documentation is the third area.

If you use the tool and find a confusing step, open an issue. If a command output is unclear, say so. If a setup step fails on Linux, Windows, or a different Python environment, include the exact output.

Good bug reports make the tool better faster.

Useful issue format:

Command:
Expected:
Actual:
OS:
Node version:
Python version:
Claude Code version:
ATS platform, if relevant:
Redacted screenshot or terminal output:

That gives enough context for someone to reproduce the issue without guessing.

Known Boundaries

career-agent handles repeated work. It does not remove your responsibility.

It does not submit applications.

It does not answer EEO or voluntary self-identification fields.

It does not enter passwords or credentials.

It does not guarantee every ATS page will work.

It does not make weak applications strong.

It does not replace your judgment about whether a role is worth applying to.

That last point matters. A tool can help you apply more consistently, but it cannot decide what kind of work you should do next. It cannot know which company culture you want. It cannot know whether a role is worth your time. It can help with the structure, not the life decision.

There is also a documentation boundary around /source.

The command is designed to find and verify open roles, score fit, and return application URLs. But any sourcing workflow should still be checked against the company career page. Roles close. Pages change. Sponsorship language can be unclear.

Use /source to build a shortlist. Do not treat it as a final truth source.

Troubleshooting Notes

If npx @nextwebb/career-agent fails, check Node first:

node --version

You need Node 18 or newer.

If setup says Python is missing, check:

python3 --version
python3.11 --version

The installer looks for Python 3.10 or newer. On some machines, python3 may point to an older version while python3.11 exists.

If reportlab is missing, install it:

python3 -m pip install reportlab

If claude --version fails, install or fix Claude Code CLI before continuing.

If the slash commands do not appear after install, restart Claude Code and check the plugin settings.

If /apply cannot control Chrome, check that Claude in Chrome is installed and connected to Claude Code. Again, signing in to Claude.ai alone is not the same thing.

Contributing

The code is open source and MIT licensed.

The repo is here:

github.com/nextwebb/career-agent

The live docs are here:

nextwebb.github.io/career-agent 📌

The npm package is here:

@nextwebb/career-agent 📦

The main contribution area is ATS support.

Greenhouse, Lever, and Workable are the current supported platforms. Ashby, iCIMS, SmartRecruiters, Taleo, and BambooHR are good candidates for future support.

If you want to contribute, open an issue first. Include the ATS platform, a sample public application URL if you can share one, what fields were missed, and what behavior you expected.

Good issues make automation work easier.

Wrapping Up

career-agent gives you a local Claude Code workflow for preparing job applications.

It helps you build a profile, source roles, create role configs, generate tailored PDFs, fill supported ATS forms, and track your pipeline.

It does not remove the need to think.

That is the point.

Use it for the repeated work. Slow down where accuracy, consent, and judgment matter.

Star the repo: github.com/nextwebb/career-agent ⭐

Read the docs: nextwebb.github.io/career-agent 📌

npm package: @nextwebb/career-agent 📦

Upvote on Product Hunt if this was useful: career-agent 🚀

If you find a bug or want to add ATS support for Ashby, iCIMS, SmartRecruiters, Taleo, or another platform, open an issue or send a PR.

If you have any inquiries or wish to gain additional knowledge, please get in touch with me on GitHub, Twitter/X, or LinkedIn. Kindly show your support by leaving a thumbs up 👍, a comment 💬, and sharing this article with your network 😊.

Hey there! If you’ve been curious about data engineering, this guide will help you understand the basics and walk you through practical examples. Whether it’s setting up storage, processing data, automating workflows, or monitoring systems, I’ll keep it simple, relatable, and fun! 😊

What is Data Engineering?

Data engineering involves organizing, processing, and automating data workflows to make raw data useful for analysis and decision-making. Here’s what we’ll cover:

1. Storage: Where and how data lives.
2. Processing: Cleaning and transforming raw data.
3. Automation: Running workflows seamlessly.
4. Monitoring: Ensuring everything runs smoothly.

Let’s dive into each step! 🚀

Setting Up Your Environment and Prerequisites

Before we start, let’s set the stage for what you’ll need:

1. Environment:

   • A Unix-based system (MacOS) or Windows Subsystem for Linux (WSL).
   • Python installed (preferably version 3.11).
   • PostgreSQL installed and running locally.

2. Prerequisites:

   • Basic Command Line Knowledge: We’ll use terminal commands for installations and setups.
   • Python Basics: Familiarity with Python syntax.
   • Administrative Access: To install and configure software.

3. Architectural Overview:

The diagram above provides a high-level view of how the component's workflow interacts; This modular approach allows each tool to do what it’s best: Airflow for orchestration, Spark for distributed data processing, and PostgreSQL for structured data storage.

4. Install Necessary Tools:

   • PostgreSQL:

     brew update
     brew install postgresql
     

   • PySpark:

     brew install apache-spark
     

   • Airflow:

     python -m venv airflow_env
     source airflow_env/bin/activate  # macOS/Linux
     #https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt
     pip install "apache-airflow[postgres]==" --constraint 
     "https://raw.githubusercontent.com/apache/airflow/constraints-2.10.4/constraints-3.11.txt"
     
     airflow db migrate #Initialize the Database
     

     

Now that we’re ready, let’s dive into each component! 💻

1. Storage: Databases and File Systems

Data storage forms the foundation of data engineering. Storage can be broadly categorized into:

• Databases:

  • Efficiently organized.
  • Features include search, replication, and indexing.
  • Examples:
    • SQL Databases: Structured data (e.g., PostgreSQL, MySQL).
    • NoSQL Databases: Schema-less (e.g., MongoDB, Redis).

• File Systems:

  • Suitable for unstructured data.
  • Limited features compared to databases.

Setting Up PostgreSQL

1. Start the Service:

   brew services start postgresql
   

   

2. Create a Database, Connect to Database and Create Table:

   CREATE DATABASE sales_data;
   \c sales_data
   CREATE TABLE sales (
       id SERIAL PRIMARY KEY,
       item_name TEXT,
       amount NUMERIC,
       sale_date DATE
   );
   

3. Insert Sample Data:

   INSERT INTO sales (item_name, amount, sale_date)
   VALUES ('Laptop', 1200, '2024-01-10'),
          ('Phone', 800, '2024-01-12');
   

   

Now your data is safely stored in PostgreSQL! 🏠

2. Processing: PySpark and Distributed Computing

Processing frameworks enable you to transform raw data into actionable insights. Apache Spark, with its distributed computing model, is one such framework that’s widely used.

• Batch and Stream Processing:
  • Batch: Processes data in chunks.
  • Stream: Processes data in real-time.
• Tools: Apache Spark, Flink, Kafka, Hive.

Processing Data with PySpark

1. Install Java and PySpark:

   brew install openjdk@11 && brew install apache-spark
   

2. Load Data from a CSV: Create a sales.csv file:

   id,item_name,amount,sale_date
   1,Laptop,1200,2024-01-10
   2,Phone,800,2024-01-12
   

   Python script to load and process:

   from pyspark.sql import SparkSession
   
   spark = SparkSession.builder.appName("DataProcessing").getOrCreate()
   
   # Load CSV File
   df = spark.read.csv("sales.csv", header=True, inferSchema=True)
   df.show()
   

   

3. Filter High-Value Sales:

   # Filter sales above $1000
   high_value_sales = df.filter(df["amount"] > 1000)
   high_value_sales.show()
   

   

   

4. Setup Postgres DB driver: If you don’t find the driver, download it directly from the PostgreSQL website: curl -O https://jdbc.postgresql.org/download/postgresql-42.6.0.jar This will download the driver to your current working directory.

Before running the script, verify the .jar file path:

ls -l /path/to/postgresql-42.6.0.jar

Once you locate the driver file, use its full path in your script.

     #Add the PostgreSQL JDBC driver path
     spark = SparkSession.builder \
    .appName("DataProcessing") \
    .config("spark.jars", "/path/to/postgresql-42.6.0.jar") \
    .getOrCreate()

5). Save Processed Data Back to PostgreSQL:

     #Add the PostgreSQL JDBC driver path
     spark = SparkSession.builder \
    .appName("DataProcessing") \
    .config("spark.jars", "/path/to/postgresql-42.6.0.jar") \
    .getOrCreate()

   high_value_sales.write.format("jdbc").options(
       url="jdbc:postgresql://localhost:5432/sales_data",
       driver="org.postgresql.Driver",
       dbtable="high_value_sales",
       user="your_user",
       password="your_password"
   ).mode("append").save()

Great work! You’ve processed data using Spark. 🚀

3. Automation: Airflow

Automation helps in managing workflows by setting up dependencies and schedules. Tools like Airflow, Oozie, and Luigi simplify this process.

Automating ETL with Airflow

1. Initialize apache Airflow:

   airflow db migrate
   
   airflow users create \
       --username admin \
       --firstname Admin \
       --lastname User \
       --role Admin \
       --email admin@example.com
   airflow webserver &
   

   

   

2. Create a Workflow (DAG):

   from airflow import DAG
   from airflow.operators.bash import BashOperator
   from datetime import datetime
   
   default_args = {
     'owner': 'airflow',
     'start_date': datetime(2024, 1, 1),
     'email': ['alerts@example.com'],
     'email_on_failure': True,
   }
   
   with DAG('spark_etl_pipeline', default_args=default_args, schedule_interval='@daily') as dag:
    #Update with the absolute path to your script
    spark_etl_task = BashOperator(
        task_id='spark_etl',
        bash_command='python3 /absolute/path/to/sales.py',  # Replace with the actual path to sales.py
    )
   
    #Example additional tasks for data verification
    verify_task = BashOperator(
        task_id='verify_results',
        bash_command='echo "Verification complete!"'
    )
   
    #Define the workflow
    spark_etl_task >> verify_task
   

   In this DAG: The schedule_interval='@hourly' ensures the ETL pipeline runs every hour. The bash_command executes the PySpark ETL script (sales.py). Email alerts notify the owner in case of failures.

3. Monitor Your Workflow:

   Move the DAG file to Airflow’s dags/ directory:

mv spark_etl_pipeline.py ~/airflow/dags/

Restart Airflow services:

airflow scheduler
airflow webserver

Check the spark_etl_pipeline DAG, and ensure it is enabled. Trigger it manually to validate.

Open the Airflow UI at http://localhost:8080. Check task statuses and logs.

4. Monitoring

Monitoring ensures pipelines run reliably. You can use tools like Airflow’s alerting system or integrate with Grafana and Prometheus.

From the Airflow UI

• Task Status: Check the Graph View at http://localhost:8080 to monitor task states:
  • Green: Success
  • Red: Failure
  • Yellow: Running
  • Grey: Queued/Skipped
• Logs: Click on a task instance to view logs for debugging.

Wrapping Up

Congrats! 🎉 You’ve learned how to:

• Set up storage with PostgreSQL.
• Process data using PySpark.
• Automate workflows with Airflow.
• Monitor systems for reliability.

Data Engineering as a Career Focus

Data engineering is the backbone of modern data workflows. It focuses on building systems that collect, store, and process data efficiently, enabling teams like analysts, data scientists, and business stakeholders to extract value from it.

While it primarily involves skills like coding, system design, and infrastructure management, data engineering often intersects with adjacent fields: backend development (e.g., APIs, database schema design), DevOps (e.g., CI/CD pipelines, infrastructure as code), and data operations (e.g., managing ETL workflows and ensuring data reliability).

Whether you're orchestrating workflows with tools like Apache Airflow, processing massive datasets with Apache Spark, or managing databases like PostgreSQL, your role is pivotal in enabling data-driven decisions.

Start small, explore, and grow your expertise! 💪

If you have any inquiries or wish to gain additional knowledge, please get in touch with me on GitHub, Twitter, or LinkedIn. Kindly show your support by leaving a thumbs up 👍, a comment 💬, and sharing this article with your network 😊.

References to Explore Further:

• Apache Spark Documentation
• Airflow Official Site
• PostgreSQL Documentation
• Constraints file for each version of Python that Airflow supports

If you’ve ever had a Python script fail to locate files in S3, chances are, case sensitivity might have been the issue.

Why Does Case Sensitivity Matter?

Let’s say your data processing pipeline dynamically generates S3 paths based on inputs from multiple teams. One team might upload to my-bucket/data/, while another uses my-bucket/Data/. Without a strategy to handle case mismatches, your pipeline could skip files or fail altogether, causing inefficiencies and delays.

How to Handle Case Sensitivity in Python

Here’s how you can address this:

1. Normalize Paths:

   • Standardize paths to lowercase (or a consistent format) during both upload and access.

2. Verify Object Keys:

   • Use AWS SDK methods like list_objects_v2 to confirm the existence of object keys.

3. Implement Error Handling:

   • Design scripts to handle exceptions KeyError and log issues for debugging.

Code Example: Listing Objects Safely

Below is a Python script to list objects in an S3 bucket while addressing case sensitivity:

import boto3

def normalize_s3_path(bucket, prefix):
    """
    Normalize and validate S3 paths to handle case sensitivity.

    Args:
        bucket (str): Name of the S3 bucket.
        prefix (str): Prefix (folder path) in the bucket.

    Returns:
        list: Canonical paths matching the prefix.
    """
    s3 = boto3.client('s3')
    response = s3.list_objects_v2(Bucket=bucket, Prefix=prefix.lower())

    if 'Contents' not in response:
        raise ValueError(f"Path '{prefix}' not found. Check case sensitivity.")

    return [obj['Key'] for obj in response['Contents']]

# Example usage
bucket_name = "my-bucket"
s3_path = "Data/File.txt"

try:
    files = normalize_s3_path(bucket_name, s3_path)
    print("Canonical paths found:", files)
except ValueError as e:
    print("Error:", e)

This script ensures that your workflow identifies object keys, regardless of mismatched cases in input paths.

Real-World Scenario

In many data processing workflows, case mismatches in file paths can lead to missing or duplicated records. For instance, a team processing customer records stored in S3 noticed recurring errors due to inconsistent casing in object keys. By implementing strategies like normalizing paths and validating keys, they were able to significantly reduce these issues and improve the reliability of their data pipelines.

Key Takeaways

• Standardize: Use consistent casing for all S3 paths.

• Validate: Leverage AWS SDKs to confirm object key existence.

• Handle Errors Gracefully: Design scripts to log and report mismatched paths.

By addressing case sensitivity early in your workflow, you can prevent costly errors and build more resilient systems.

What About You?

Have you faced challenges with case sensitivity in S3? Share your experiences in the comments or connect with me to discuss more strategies for optimizing cloud workflows!

If you have any inquiries or wish to gain additional knowledge, please get in touch with me on GitHub, Twitter, or LinkedIn. Kindly show your support by leaving a thumbs up 👍, a comment 💬, and sharing this article with your network 😊.

References

• Naming Amazon S3 Objects: AWS Documentation

🎯 This article targets developers, architects, and AWS users who want to better understand the differences between CloudWatch Events and EventBridge and choose the most suitable service for triggering AWS Glue Jobs and Step Functions workflows.

🌟 Understanding Event-Driven Computing on AWS

According to the AWS documentation, event-driven computing is a computing model in which an application responds to events, which can be generated either by a user, a software system, or a hardware device. AWS offers several services for event-driven computing, including Amazon CloudWatch Events and Amazon EventBridge.

💻 CloudWatch Events vs. EventBridge: Key Differences

When choosing between CloudWatch Events and EventBridge, it is essential to understand their differences.

👴 Maturity: CloudWatch Events is an older service compared to EventBridge. Suppose you already use CloudWatch Events in your environment and only need basic event routing and triggering for Glue Jobs and Step Functions. In that case, it might make sense to continue using CloudWatch Events.

🔌 Flexibility and Integration: EventBridge offers more flexibility and integration capabilities, especially when working with custom events or third-party SaaS providers. If your Glue Jobs or Step Functions workflows require complex event filtering, transformation, or integration with external services, EventBridge is a better choice.

🚌 Event Bus and Event Routing: EventBridge uses an event bus to manage and route events, providing better control and visibility over event routing and delivery. This feature can be beneficial when coordinating multiple event sources and targets or needing more granular control over event processing.

🚀 Scalability: EventBridge is designed to handle large volumes of events and offers better scaling capabilities than CloudWatch Events. If you anticipate high event throughput for your Glue Jobs or Step Functions workflows, consider using EventBridge.

🌩️ Using CloudWatch Events with AWS Glue Jobs

AWS Glue is a fully-managed extract, transform, and load (ETL) service that enables you to move data between data stores. In addition, you can use CloudWatch Events to trigger Glue jobs when specific events occur in your AWS environment.

👉 Here's an example of how you can use a CloudWatch Events rule to trigger a Glue job when a new object is created in an S3 bucket:

{
   "source":[
      "aws.s3"
   ],
   "detail-type":[
      "AWS API Call via CloudTrail"
   ],
   "detail":{
      "eventName":[
         "PutObject"
      ],
      "requestParameters":{
         "bucketName":[
            "my-bucket"
         ]
      }
   }
}

🌩️ Using EventBridge with AWS Step Functions

AWS Step Functions is a fully-managed service that enables you to coordinate multiple AWS services into serverless workflows. You can use EventBridge to trigger Step Functions workflows when specific events occur in your AWS environment.

👉 Here's an example of how you can use an EventBridge rule to trigger a Step Functions workflow when an EC2 instance is stopped:

{
   "source":"aws.ec2",
   "detail-type":"EC2 Instance State-change Notification",
   "detail":{
      "state":"stopped"
   }
}

🔍 Prerequisites and Best Practices

Before using CloudWatch Events or EventBridge with Glue jobs or Step Functions, consider the following prerequisites and best practices:

✅ Ensure that your AWS environment is configured correctly and you have the necessary permissions to create and manage CloudWatch Events or EventBridge rules and targets.

🎯 Define explicit event schemas and rules that match your specific use cases and requirements.

🔎 Use filtering and transformation capabilities to limit the number of events processed and to optimize the event payload for downstream services.

🧪 Test and validate your event-driven workflows to ensure that they are working as expected, and use monitoring tools like CloudWatch to monitor the performance and health of your workflows.

In conclusion, choosing between CloudWatch Events and EventBridge depends on the specific requirements of your application. For example, if you only need basic event triggering and routing, CloudWatch Events might be sufficient. However, if you require more advanced event processing, integration, and scalability, EventBridge is better for managing your AWS Glue Jobs and Step Functions workflows.

Remember to follow best practices and optimize your event-driven workflows for performance and scalability. By choosing exemplary event-driven service and best practices, you can build scalable and resilient applications on AWS.

If you have any inquiries or wish to gain additional knowledge, please get in touch with me on GitHub, Twitter, and LinkedIn. Moreover, kindly show your support by leaving a thumbs up 👍, a comment 💬, and sharing this article with your network 😊.

References:

AWS Documentation, Amazon EventBridge. https://aws.amazon.com/eventbridge/

AWS Documentation, What is AWS Glue? https://aws.amazon.com/glue/

AWS Documentation, What is AWS Step Functions? https://aws.amazon.com/step-functions/

Looking to streamline your complex workflows and keep everything on track? Look no further than state machines - a powerful tool for precision and ease in process management. 💪🏼 And when it comes to implementing state machines, AWS Step Functions has covered you with a simple JSON-based language and visual representation. 🌟

This article will introduce you to state machines and provide a walkthrough for creating a basic example using AWS Step Functions. You can say goodbye to manual workflow management with automated optimization and enhanced efficiency. 🤖 Are you looking to improve how you handle challenging tasks? Let's get started now! 💻

Prerequisites 🧑‍💻

• Basic understanding of programming concepts

• Some knowledge of AWS services.

• Some knowledge of lambdas

The Experiment 😉

A state machine is a fundamental concept that helps define how a system behaves. It comprises a set of states and the transitions between them. 🚦 These transitions are caused by specific events and follow particular conditions or rules. This feature makes it possible to model different types of processes, like order fulfillment, data pipelines, and machine learning models.

For example, you have an e-commerce website and want to build a state machine for order fulfillment. Your state machine could have states such as "Order Received," "Payment Received," "Order Shipped," and "Order Delivered." 📦 The state machine triggers transitions between these states when it receives events such as "Payment Received" or "Order Shipped." 💳

Now, let's talk about AWS Step Functions. This service allows you to build, run, and visualize state machines using a simple JSON-based language. 🚀 You can use it to model your application's workflow and see a visual representation of the states and transitions. Additionally, Step Functions provides automatic retries and error handling, parallel execution, and the ability to integrate with other AWS services. 🌟

Here's an example of a simple state machine template in JSON for AWS Step Functions:

{
  "Comment": "A simple example of a state machine",
  "StartAt": "Order Received",
  "States": {
    "Order Received": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:REGION:ACCOUNT_ID:function:OrderReceived",
      "Next": "Payment Received"
    },
    "Payment Received": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:REGION:ACCOUNT_ID:function:PaymentReceived",
      "Next": "Order Shipped"
    },
    "Order Shipped": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:REGION:ACCOUNT_ID:function:OrderShipped",
      "Next": "Order Delivered"
    },
    "Order Delivered": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:REGION:ACCOUNT_ID:function:OrderDelivered",
      "End": true
    }
  }
}

This template defines a state machine with four states: "Order Received," "Payment Received," "Order Shipped," and "Order Delivered." Each state corresponds to a task implemented by a Lambda function. The state machine starts at "Order Received" and moves through the states in the order specified by the "Next" field. The "Order Delivered" state is an end state, as indicated by the "End" field set to true.

Let's go through creating and running a state machine using Step Functions.

1. 🏭 Go to the AWS Step Functions Console and create a new state machine.

   

2. 🔨 Define your states and the transitions between them using the JSON-based language. You can use the template above as a starting point and customize it to fit your specific use case.

You will see the visual representation of the states and transitions and ensure everything works as expected.

1. ✅ After you have reviewed and customized the state machine (ignore the squiggly lines above, the lambda ARNs here are just for testing) to meet your requirements,

You can configure the default settings and create an execution role that provides the necessary permissions for the state machine to interact with the resources defined in your workflow. Once confirmed that everything is working as expected, you can initiate the execution of your state machine by triggering its start state.

1. 🛠️ Create and Test your state machine using the Step Functions Console.

2. 🚀 It's essential to remember that this is a simplified example, and state machines can be much more complex depending on your use case. You can add other states, such as "Choice" or "Wait" states, and specify input and output paths, error handling, and retry options.

In conclusion,

State machines are a powerful tool for managing complex workflows, and AWS Step Functions provides a simple and efficient implementation. You can quickly build and state machines using the JSON-based language and the visual representation provided by the console.

🤓 However, remember that this is a complex topic that requires a deep understanding of the concept and the use of AWS Step Functions.

Feel free to contact me on GitHub, Twitter, and LinkedIn if you have any inquiries or wish to gain additional knowledge. Moreover, kindly show your support by leaving a thumbs up 👍, a comment 💬, and sharing this article with your network 😊.

References

Some helpful resources to get started include the following:

• AWS Step Functions Getting Started Guide: https://docs.aws.amazon.com/step-functions/latest/dg/getting-started.html

• AWS Step Functions Developer Guide: https://docs.aws.amazon.com/step-functions/latest/dg/welcome.html

• AWS Step Functions Sample State Machines: https://docs.aws.amazon.com/step-functions/latest/dg/sample-state-machines.html

🚀 This tutorial will look at deploying Prometheus monitoring on AWS using Ansible for configuration management and Terraform for infrastructure as code. We will be deploying Prometheus on EC2 instances and behind a load balancer.

🧑‍💻 Prerequisites

• An AWS account

• Basic knowledge of AWS and Terraform

• Basic knowledge of Ansible

• A text editor

🚀 Project Overview

We will deploy Prometheus monitoring on three EC2 instances behind a load balancer. We will use Terraform to create the EC2 instances, load balancer, and Ansible to configure the instances and deploy Prometheus.

Here is the project directory map:

📁 prometheus-ansible-terraform/

├── 📄 inventory.ini

├── 📄 main.tf

├── 📄 playbook.yml

├── 📄 prometheus.yml

├── 📄 site.yml

Here are the steps we will take:

1. Deploy EC2 instances and a load balancer using Terraform

2. Configure EC2 instances using Ansible

3. Install and configure Prometheus on the EC2 instances

🔧 Terraform Configuration

1. Create a new directory for your Terraform configuration and create a main.tf file.

2. Inside the main.tf file, define the provider as AWS, and specify the region:

    pythonCopy codeprovider "aws" {
      region = "us-east-1"
    }
   

3. Create an EC2 instance and load balancer resource in the main.tf file:

    resource "aws_instance" "prometheus" {
      ami           = "ami-0c94855ba95c71c99"
      instance_type = "t2.micro"
      count         = 3
   
      tags = {
        Name = "prometheus"
      }
    }
   
    resource "aws_lb" "prometheus" {
      name = "prometheus"
      internal = false
   
      listener {
        lb_port = 80
        protocol = "HTTP"
   
        instance_port = 9090
        instance_protocol = "HTTP"
      }
    }
   

   This Terraform configuration will create three EC2 instances and a load balancer. The EC2 instances will be running the Amazon Machine Image (AMI) with ID ami-0c94855ba95c71c99, which is a base Amazon Linux image. We will use this as the base for our Prometheus deployment.

   The load balancer will listen on port 80 and forward traffic to port 9090 on the EC2 instances.

4. Run the following command to initialize the Terraform project:

    terraform init
    terraform apply
   

🛠️ Ansible Configuration

1. Create a new directory for your Ansible configuration and create a inventory.ini file. Inside the file, specify the EC2 public IP addresses provisioned by the terraform code:

    [prometheus]
    ec2-x-x-x-x.compute-1.amazonaws.com
    ec2-y-y-y-y.compute-1.amazonaws.com
    ec2-z-z-z-z.compute-1.amazonaws.com
   

2. Create a site.yml file with the following playbook that configures the EC2 instances:

    - hosts: prometheus
      become: true
      tasks:
        - name: Install necessary packages
          yum:
            name:
              - wget
              - git
              - unzip
              - java-1.8.0-openjdk
            state: present
   
        - name: Download and install node_exporter
          get_url:
            url: https://github.com/prometheus/node_exporter/releases/download/v1.0.1/node_exporter-1.0.1.linux-amd64.tar.gz
            dest: /tmp/node_exporter.tar.gz
          become_user: ec2-user
   
        - name: Extract node_exporter
          unarchive:
            src: /tmp/node_exporter.tar.gz
            dest: /usr/local/bin/
            remote_src: true
            copy: no
          become_user: root
   
        - name: Configure node_exporter
          template:
            src: templates/node_exporter.service.j2
            dest: /etc/systemd/system/node_exporter.service
            owner: root
            group: root
            mode: '0644'
          notify:
            - reload systemd
   

3. Create a prometheus.yml file that defines the Prometheus configuration:

   global:
     scrape_interval: 5s

   scrape_configs:
     - job_name: ec2-instances
       ec2_sd_configs:
         - region: us-east-1
           port: 9100
       relabel_configs:
         - source_labels: [__meta_ec2_tag_Name]
           target_label: instance
         - source_labels: [__meta_ec2_private_ip]
           target_label: ip

     - job_name: load-balancers
       ec2_sd_configs:
         - region: us-east-1
           port: 9100
           filters:
             - name: "tag:Component"
               values: ["LoadBalancer"]
       relabel_configs:
         - source_labels: [__meta_ec2_tag_Name]
           target_label: load_balancer
         - source_labels: [__meta_ec2_private_ip]
           target_label: ip

This configuration file specifies two scrape jobs: one for EC2 instances and one for load balancers. The EC2 job uses EC2 service discovery to dynamically discover all running instances and scrape their metrics using the Node Exporter on port 9100.

The load balancer job filters by instances with a Component tag set to LoadBalancer.

Next, let's create an Ansible playbook to install and configure Prometheus on our EC2 instances. We'll create a file named playbook.yml in the same directory as our site.yml playbook.

1. Create a new Ansible playbook in a file named playbook.yml that installs and configures Prometheus:

    - name: Install Prometheus
      hosts: prometheus
      become: true
      gather_facts: true
   
      tasks:
        - name: Install Prometheus
          apt:
            name: prometheus
            state: present
   
        - name: Copy Prometheus configuration file
          template:
            src: prometheus.yml
            dest: /etc/prometheus/prometheus.yml
   
        - name: Start Prometheus service
          systemd:
            name: prometheus
            state: started
            enabled: yes
   

2. Run the site.yml playbook to configure the EC2 instances:

    ansible-playbook -i inventory.ini site.yml
   

3. Run the playbook.yml playbook to install and configure Prometheus:

    ansible-playbook -i inventory.ini playbook.yml
   

Conclusion🎉

In this tutorial, we learned how to use Terraform to deploy an infrastructure on AWS that includes EC2 instances, an Application Load Balancer, and a Prometheus server. We also learned how to use Ansible to configure and manage the Prometheus server. With these tools, we can automate the deployment and management of our infrastructure and ensure that our Prometheus monitoring system is up and running smoothly.

This is a basic example, but you can do many other things with Terraform and Ansible to manage your infrastructure and automate your workflows. By mastering these tools, you can build robust and scalable cloud-based systems that meet the needs of your business or organization.

Don't hesitate to reach out to me on GitHub, Twitter, and LinkedIn if you have any questions or want to learn more. And don't forget to give a thumbs up 👍, leave a comment 💬 and share this article with your network 😊.

References:

• Terraform documentation: https://www.terraform.io/docs/index.html

• Ansible documentation: https://docs.ansible.com/

• Prometheus documentation: https://prometheus.io/docs/introduction/overview/

• AWS documentation: https://aws.amazon.com/documentation/

Introduction📝

When it comes to building serverless applications, several options are available for coordinating and executing tasks. AWS Step Functions 🤖 and the Chalice framework 🛠️ are two popular choices, but Python also provides built-in support for asynchronous programming through the async and await keywords 🔑. In this article, we will compare these three options and explore when to use each one to build scalable and maintainable serverless applications using Python 🐍.

Prerequisites 🧑‍💻

This article assumes that you have the following:

• Some essential knowledge of Python

• Know what is AWS step functions and lambda

• Are you familiar with serverless applications

Discussions 💬

AWS Step Functions is a service that allows you to create and run workflows that coordinate multiple AWS services, such as Lambda functions, DynamoDB tables, and SQS queues. These workflows, known as state machines, are defined using the JSON-based Amazon States Language. Step Functions provides a powerful way to visualize and understand the flow of your application, which can help with troubleshooting and debugging 🔍. However, it does add a layer of complexity, as you need to manage the state machine and its associated resources 🤔.

For example, consider a simple workflow that processes an image file uploaded to an S3 bucket. The workflow could consist of the following steps:

1). First, a Lambda function is triggered by the S3 event and resizes the image.📷

import json
import boto3

s3 = boto3.client('s3')

def resize_image(event, context):
    bucket = event['Records'][0]['s3']['bucket']['name']
    key = event['Records'][0]['s3']['object']['key']
    # code to resize image
    s3.put_object(Bucket=bucket, Key=key, Body=resized_image)

2). Another Lambda function is triggered by the first one and applies the watermark to the image.💦

import json
import boto3

s3 = boto3.client('s3')

def apply_watermark(event, context):
    bucket = event['Records'][0]['s3']['bucket']['name']
    key = event['Records'][0]['s3']['object']['key']
    # code to apply watermark
    s3.put_object(Bucket=bucket, Key=key, Body=watermarked_image)

3). A final Lambda function that stores the processed image in a different S3 bucket.📁

import json
import boto3

s3 = boto3.client('s3')

def store_image(event, context):
    bucket = event['Records'][0]['s3']['bucket']['name']
    key = event['Records'][0]['s3']['object']['key']
    # code to get the processed image
    s3.put_object(Bucket='processed-images', Key=key, Body=processed_image)

Using step functions 🚀🧭

With Step Functions, you can define these steps in a state machine 📈 and have them executed in the specified order, with error handling 🤕 and retries built-in. This can significantly simplify the development process 🛠️ and make it easier to manage the application 💼.

Here is an example of a simple state machine for the Step Functions workflow:

{
"Comment": "A simple workflow that processes an image file uploaded to an S3 bucket",
"StartAt": "Resize Image",
"States": {
  "Resize Image": {
    "Type": "Task",
    "Resource": "arn:aws:lambda:REGION:ACCOUNT_ID:function:resize_image",
    "Next": "Apply Watermark"
  },
  "Apply Watermark": {
    "Type": "Task",
    "Resource": "arn:aws:lambda:REGION:ACCOUNT_ID:function:apply_watermark",
    "Next": "Store Image"
  },
  "Store Image": {
    "Type": "Task",
    "Resource": "arn:aws:lambda:REGION:ACCOUNT_ID:function:store_image",
    "End": true
  }
}
}

The above code block is an example of a state machine for the Step Functions workflow. It defines a simple workflow that processes an image file uploaded to an S3 bucket. The workflow consists of three steps: "Resize Image", "Apply Watermark", and "Store Image". Each step is represented by a state in the state machine and is executed in the specified order.

Using Chalice and Python Async methods 🔥🐍💻

It provides a simple, intuitive way to define and deploy Lambda functions 💻 and configure the associated resources, such as API Gateway endpoints 🌐 and IAM roles 🔒.

Chalice makes it easy to start serverless development 🛠, as it abstracts away many of the complexities of working with AWS services directly.

Here is an example of how the same example can be coordinated/invoked within Chalice and Python async methods:

from chalice import Chalice, Response
import json
import boto3
import asyncio

app = Chalice(app_name='image-processing')
s3 = boto3.client('s3')

@app.lambda_function()
def resize_image(event, context):
    bucket = event['Records'][0]['s3']['bucket']['name']
    key = event['Records'][0]['s3']['object']['key']
    # code to resize image
    s3.put_object(Bucket=bucket, Key=key, Body=resized_image)
    apply_watermark.apply(event, context)

@app.lambda_function()
async def apply_watermark(event, context):
    await asyncio.sleep(1)
    bucket = event['Records'][0]['s3']['bucket']['name']
    key = event['Records'][0]['s3']['object']['key']
    # code to apply watermark
    s3.put_object(Bucket=bucket, Key=key, Body=watermarked_image)
    store_image.apply(event, context)

@app.lambda_function()
async def store_image(event, context):
    await asyncio.sleep(1)
    bucket = event['Records'][0]['s3']['bucket']['name']
    key = event['Records'][0]['s3']['object']['key']
    # code to get the processed image
    s3.put_object(Bucket='processed-images', Key=key, Body=processed_image)

In the above example, the resize_image function is invoked by the S3 event, and once it completes the execution, the apply_watermark function is called, and the same goes for the last function. Chalice framework handles the invocation of these functions and orchestrates them together.

🤓 If you're looking to build a serverless application using Python, you have several options! One popular option is AWS Step Functions, which allows you to easily coordinate multiple AWS services and simplify the development process for complex applications. Another option is the Chalice framework, which provides a simple and intuitive way to define and deploy Lambda functions and configure resources such as API Gateway endpoints and IAM roles.

💻 But perhaps the most exciting option is using Python's built-in support for asynchronous programming through the async and await keywords. This allows you to write non-blocking code, handling multiple tasks simultaneously, rather than waiting for one task to complete before starting the next. You have more control over the flow of your application and can take advantage of built-in libraries such as asyncio and aiohttp. However, it requires more expertise and experience with asynchronous programming, and you will need to handle things like scaling and error handling yourself.

🤔 Ultimately, the best choice will depend on the specific requirements of your application and your team's expertise. The Chalice framework may be the best choice if you build a simple application and want to start quickly. If you need more control over the flow of your application and are comfortable with asynchronous programming, then Python async methods may be a better fit. If you are building a complex application with multiple steps and decision points, AWS Step Functions can help simplify the development process.

💡 It's also worth noting that these three approaches are not mutually exclusive, and you can combine them to build more complex applications. For example, you could use Step Functions to coordinate the flow of your application and use Chalice and Python async methods to handle the logic of your application.

🚀 So, what are you waiting for? Take the first step towards building your dream serverless application using Python today! 🚀

Conclusion🤔

Building serverless applications with Python is a breeze with AWS Step Functions 🚀, the Chalice framework 🛠️, and Python async methods 💻. Each of these tools has its unique strengths and can be used to achieve different goals.

AWS Step Functions makes it super simple to coordinate and execute multiple Lambda functions in a specific order📝. At the same time, the Chalice framework takes the complexity out of deploying and configuring Lambda functions and other resources🌎. With Python async methods, you can build ultra-efficient and scalable applications that can handle high levels of concurrency🔥. Whether you're just getting started with serverless or are a seasoned pro, these tools are the perfect choice for your next project! So what are you waiting for? Give them a try and see the magic for yourself🌟🚀.

Don't hesitate to reach out to me on GitHub, Twitter, and LinkedIn if you have any questions or want to learn more. And don't forget to give a thumbs up, leave a comment and share this article with your network 😊.

References

• AWS Step Functions: https://aws.amazon.com/step-functions/

• Chalice: https://aws.github.io/chalice/

• Python's built-in support for asynchronous programming: https://docs.python.org/3/library/asyncio.html

But first, what is Ansible? 🤔 Ansible is an open-source automation platform that allows you to automate configuration management, application deployment, and orchestration tasks. It is simple to use and does not require an agent or additional custom security infrastructure, making it an ideal choice for automation. This article will walk you through the process of quickly automating Docker container creation using Ansible.

Prerequisites

This article assumes that you have 😉:

• Some essential experience with Docker

• Know what a playbook is and how to create one in Ansible

• Are familiar with Jenkins and how to set up a workflow

Let's dive into a real-world use case to see how Ansible can make Our lives easier. 🌎

A load balancer is like a 🚨 traffic cop directing people to the right server when they want to use a web app. To ensure the app works for everyone, we have to ensure each server has the correct version of the app and the right dependencies installed. We can use a tool like Ansible to set everything up consistently on all the servers. 💻💾🧑💻💾🧑💻💾🧑

Now, look at code samples to see how we can use Ansible to create Docker containers. 📝

First, we need to install the Docker python module and the ansible Docker module:

pip install docker
pip install ansible[docker]

Next, we can create a playbook <playbook_name>.yml that uses the docker_container Module to create a Docker container. 🐳 Here is an example of how to do this:

   - name: Create Docker container 🐳
      hosts: localhost
      tasks:
        - name: Pull image from DockerHub 📥
          docker_image:
            name: ubuntu
            source: pull
        - name: Create Docker container 📦
          docker_container:
            name: ubuntu_container
            image: ubuntu
            state: started

This playbook will pull the latest version of the Ubuntu image from DockerHub and create a Docker container based on that image. Finally, we will give the container the name ubuntu_container and start it.

You can customize the container by adding additional arguments to the docker_container Module. For example, you can specify the command to run when the container is started or mount a volume to the container.

To run the playbook, you can then run the playbook using the ansible-playbook command:

ansible-playbook <playbook_name>.yml

We can integrate this playbook into a Jenkins workflow to simplify the process. This way, we can trigger the playbook to automatically run whenever certain events occur, such as a code change or a build completion.

To integrate this playbook into a Jenkins workflow, you can do the following:

1. Install the Ansible plugin in Jenkins. It will allow you to run Ansible playbooks as part of a Jenkins job.

2. Create a new Jenkins pipeline job by defining a pipeline script in a Jenkinsfile.

3. In the pipeline definition, specify the location of your playbook file and any required variables.

   For example, below is JenkinsFile defining each stage of the pipeline job :

pipeline {
  agent {
    label 'my-ansible-agent'
  }
  stages {
    stage('Create Docker container') {
      steps {
        ansiblePlaybook(
          playbook: 'create_docker_container.yml', 
          extraVars: [
            containerName: 'ubuntu_container',
            imageName: 'ubuntu'
          ]
        )
      }
    }
  }
}

1. Run the Jenkins job to execute the playbook.

Note that you will need to have the Docker python module and the ansible Docker module installed on the Jenkins agent machine for the playbook to run successfully. You can do this by adding the following steps to your pipeline definition:

stage('Install dependencies') {
  steps {
    sh 'pip install docker'
    sh 'pip install ansible[docker]'
  }
}

It will install the required dependencies before the playbook runs.

In summary, Ansible is a powerful tool that allows you to automate the creation of Docker containers 🚀 . In addition, you can easily set up and configure multiple Docker containers by creating playbooks 📦 . So next time you create Docker containers, give Ansible a try 💪!

I hope you enjoyed reading this article and found it helpful 🤓, as much as I did writing it 😊! You can find me on GitHub, Twitter, and LinkedIn if you have any questions or want to learn more. Don't forget to show your support by giving this article a like, leaving a comment, and sharing it with your friends and followers 😊.

Here are some useful references to check out:

• Ansible documentation

• Docker python module

• Ansible Docker module

For instance, consider an API that has a "create user" endpoint. Suppose this endpoint is not idempotent, and a client inadvertently makes the same request twice. In that case, the API may create two duplicate user accounts, which could lead to data inconsistencies or other problems 🤯. On the contrary, if the endpoint is idempotent, the API will recognize that the request has already been completed and will not create a duplicate user account 😌 .

It is generally good practice to design APIs to be idempotent wherever possible, as this can make them more predictable and easier to use 💪.

Here are some reasons why idempotence is important in APIs: 🤔

1. Preventing unintended side effects: If an API endpoint is not idempotent and a client makes the same request multiple times, it could potentially cause unintended side effects, such as creating duplicate records in a database 🚫. For example, consider the following code for a non-idempotent "create user" endpoint in Node.js 💻:

    app.post("/users", (req, res) => {
      const user = req.body;
      User.create(user, (err, newUser) => {
        if (err) {
          res.status(500).send(err);
        } else {
          res.status(201).send(newUser);
        }
      });
    });
   

   If a client inadvertently makes the same request twice, this endpoint will create two identical user records in the database, which could lead to data inconsistencies 😟.

   Here is an example of how you could modify the code snippet to make it idempotent:

    app.post("/users", (req, res) => {
      const user = req.body;
      User.findOne({ email: user.email }, (err, existingUser) => {
        if (err) {
          res.status(500).send(err); 💥
        } else if (existingUser) {
          res.status(200).send(existingUser); 🟢
        } else {
          User.create(user, (err, newUser) => {
            if (err) {
              res.status(500).send(err); 💥
            } else {
              res.status(201).send(newUser); 🟢
            }
          });
        }
      });
    });
   

   This version of the code checks if the user already exists in DB. Returns existing user with 200 (OK) status code if it does. Creates new user and returns with 201 (Created) status code if it doesn't. 🔄 Prevents multiple copies of the same user in DB from identical requests💡.

2. Simplifying client code: If an API endpoint is idempotent, clients can make multiple, indistinguishable requests without having to worry about generating unintended side effects. 💡 This can simplify the client code, as the client does not need to track the system's state or handle potential error cases 💪 .

3. Improving reliability: Idempotent APIs are more reliable because clients can retry failed requests without worrying about altering the system's state. 🔄 For example, consider the following idempotent code for a "transfer funds" endpoint 💰:

    app.patch("/transfer", (req, res) => {
      const transfer = req.body;
      Transfer.findOneAndUpdate({ _id: transfer.id }, transfer, { upsert: true }, (err, updatedTransfer) => {
        if (err) {
          res.status(500).send(err);
        } else {
          res.status(200).send(updatedTransfer);
        }
      });
    });
   

   If the client's initial request fails due to a network error, it can retry the request without worrying about transferring the funds multiple times 💻 .

4. Ensuring consistency: Idempotent APIs can help ensure data consistency because clients can retry failed requests without altering the system's state. 💪 For example, consider the following idempotent code for a "create order" endpoint 🛒 :

    codeapp.post("/orders", (req, res) => {
      const order = req.body;
      Order.findOneAndUpdate({ _id: order.id }, order, { upsert: true }, (err, updatedOrder) => {
        if (err) {
          res.status(500).send(err);
        } else {
          res.status(200).send(updatedOrder);
        }
      });
    });
   

   If the client's initial request fails due to a network error, it can retry the request without worrying about creating multiple identical orders 💻 .

5. Improving performance means that if a request fails, a client can retry it without unintended consequences. It can help improve the API's performance by reducing the server's workload and making it more responsive. So if you have an idempotent API, you don't have to worry about duplicate actions or changes to the system's state when you retry a failed request. Pretty cool, right? 🤓

Queue systems can help 🤔!

In queue systems, idempotency 🤝 is used to ensure that each message 💬 in the queue is processed only once 🔁, even if the consumer receives the message multiple times 🔄. It is essential because it prevents duplicate messages 📩 from being processed and causing unintended consequences 🤯, such as creating duplicate records 📊 or triggering multiple actions 🤔.

Here is an example 💡 of how idempotency can be implemented in a queue system using BullMQ, a popular queue library for Node.js 💻

const Bull = require('bullmq');

const queue = new Bull('payment-queue');

queue.process(async (job) => {
  const paymentId = job.data.paymentId;
  const payment = await Payment.findById(paymentId);

  if (!payment) {
    // Payment not found, process the payment
    await processPayment(paymentId);
  }

  // Return a promise that resolves to remove the job from the queue
  return Promise.resolve();
});

The consumer checks the database 🗃️ to see if they have already processed a payment. If the payment is already processed, the consumer removes the job from the queue without further action. It ensures that each payment is processed only once, even if the consumer receives the message multiple times.

In conclusion, 💡

In addition to improving reliability 💪 and data consistency 🧐, idempotency in queue systems can also improve performance 📈; by reducing the number of duplicate messages that need to be processed. In addition, it can help reduce the system's overall workload, making it more efficient and scalable 📈.

Overall, idempotency is an essential property in queue systems that helps ensure the system's reliability, consistency, and performance. By designing APIs to be idempotent wherever possible, you can help to ensure that your system can handle multiple identical requests without causing unintended consequences.

Thank you for reading this article! I hope you found it helpful 🤓. If you have any questions or want to learn more, feel free to reach out to me on GitHub, Twitter, and LinkedIn. Also, don't forget to give this article a like, leave a comment, and share it with your network 😊 .

For further reading on the topic of idempotency in APIs, you may want to check out the following resources:

"HTTP Idempotent Methods" (https://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html)

"RESTful API Designing guidelines - The best practices" (https://www.slideshare.net/RestApiTutorial/restful-api-designing-guidelines-the-best-practices)

"API Idempotence - What Is It and Why Is It Important?" 🤔(https://www.moesif.com/blog/technical/api-idempotence/)

"Idempotent jobs" (https://docs.bullmq.io/patterns/idempotent-jobs)

• Stop guessing about capacity.
• Avoid huge capital investments upfront.
• Pay for only what you use.
• Scale globally in minutes.
• Deliver faster and e.t.c.

Why template your infrastructure?

With the advent of cloud computing, came several tools that enabled us to deploy the underlying infrastructure components that provide security and services to your servers by writing scripts.

Infrastructure as code (IaC) is to make operations faster by eliminating manual processes of navigating through the AWS console to provision resources and thus eliminating the slack in the process.

A code-based approach makes it easier to get more done in less time, you could introduce the version control tools, you could push templates into repositories for collaborations, semantic versioning of templates, introduce checks and reviews on the templates to reduce errors. Interesting right 😊!

Let's get started

In this tutorial, you’ll learn how to deploy infrastructure using CloudFormation script, AWS proprietary tool for Infrastructure as Code. it's used for managing, configuring, and deploying infrastructure.

Prerequisites

• Version Control: A popular recommendation is Git
• Code Editor for YAML and JSON: Vscode is a great lightweight code editor, with a plethora of plugins to customize your coding experience.
• Amazon Web Services account: visit aws.amazon.com and sign in.

Create Access key ID

• In the AWS console, Choose the region closest to your location.
• Navigate to the IAM service, click on users, create a user with "programmatic access." This allows us to use code to interact with AWS, instead of relying on mouse clicking in the AWS console.
• For IAM access, choose “administrator access.” This is just for the initial setup of your account. Afterward, you’ll want to limit access to only what you need.

Remember not to save these in your code or to check into any repositories.

After saving and creating the user, you should have this page below in the user's section:

Next, configure the AWS CLI

let's follow these steps;

• Download and install the AWS CLI tool.
• In the terminal, type aws --version: this displays the package version, thus verifying that you have the AWS CLI tool installed.
• To set up your AWS CLI, type aws configure in the terminal. Next, when prompted for the AWS credentials paste in your "Access Key ID" and "Secret access key" accordingly.
• Next, use a region closest to your location.

One way to check if your "AWS CLI" is set up properly is to try a command. You can try listing your S3 buckets with aws s3 ls. This will be blank if you have no S3 buckets. However, if you have no error message, then you’ve verified that your user has API access to communicate with AWS.

Next, write your cloudformation script

CloudFormation uses a declarative language like SQL and not an imperative language. It lets you specify what you want, without requiring you to specify how to get it.

CloudFormation abstracts the complexities of resource dependencies. So that you don’t have to specify which resource to start up before another. There are cases where you can specify that a resource depends on another resource, but ideally, you’ll let CloudFormation take care of dependencies.

Cloud formation scripts can be written in YAML and JSON file formats. but YAML is the industry preferred version that’s used for AWS.

In this example, you'll provision a VPC on AWS below. Each VPC is a fundamental layer of your infrastructure. It isolates your network environment on a public cloud.

#testcfn.yml
Description: > 
    Nextwebb / localhost
    This template deploys a VPC.
Resources:
    TestVPC:
        Type: AWS::EC2::VPC
        Properties:
            CidrBlock: 10.0.0.0/16
            EnableDnsHostnames: true

The example above provisions a resource, three parts constitute the script:

• Name: A unique name you want to give to the resource. In this case, we called it "TestVPC".

• Type: Specifies the actual hardware resource that you’re deploying. we called it "AWS VPC".

• Properties: Specifies the configuration options for your resource. You can think of these as all the drop-down menus and checkbox options that you would see in the AWS console if you were to request the resource manually. It's a good practice to consult the cloudformation docs for resource properties and know which are optional or defaults.

Just like we used above, the "CidrBlock" property specifies a range of(65,536) IP addresses, and the "EnableDnsHostnames" property Indicates whether the instances launched in the VPC get DNS hostnames.

How to run CloudFormation

• In this tutorial, we’ll assume your CloudFormation file name is called testcfn.yml, and you’re giving your stack the name "ourfirsttest".

• In the terminal, to use your yml code to request the resources, type the following in the same directory as your yml file: aws cloudformation create-stack --stack-name ourfirsttest --region us-west-2 --template-body testcfn.yml

  You may also want to use update-stack when you want to update an existing stack instead of destroying your stack and creating a new one.

Once the above command is run, we should an output response like this. Depending on the response format you chose it would be text or object containing a stackId.

A stack is a group of resources. These are the resources that you want to deploy, and that are specified in the YAML file.

Navigate to the AWS console and select the cloudformation service. You should see the name of the stack you chose and the status of the resources provisioned.

Navigate to the AWS console, you should have two VPC's. One is the default VPC that comes with your aws account and the other should be the one you created. You can differentiate them by the IP range and their Ids.

Yey! everything looks good. We successfully provisioned a VPC with cloudformation. 🚀😇

In this tutorial, you walked through setting up your AWS API user and used a cloud formation script to write a simple infrastructure. The VPC is a fundamental layer of your infrastructure. It isolates your network environment on a public cloud and other AWS resources and services operate inside of the VPC networks to provide cloud services.

You don't need to memorize the code that you need for each resource. You can find sample code in the documentation for CloudFormation for examples of how to write your CloudFormation scripts.

Thank you for the audience and I hope you found this article helpful 🤗. Feel free to reach out to me on Github, Twitter, and LinkedIn. Do drop a like, comment, and share 😌.

#Cloudcaptain 👨‍✈️

Why queue system?

A queue system would thus, efficiently cater to the increasing asynchronous operations from an increase of actions/events. thus reducing errors, improving user experience, and processing actions in the background, and executing functions or triggering an application service when it's ready.

Some tasks that can be handled by a job queue:

• Sending system-authored push notifications
• Sending system-authored emails
• communication with third applications and webhooks
• event-driven programming
• logging events, e.t.c

Necessary terminology in the context of queues;

Job - This is a task that can get pushed into a queue system, to be processed at a later stage in the application flow. the job is the Object that contains the required data for the consumer to process.

Consumer - This is the task to be carried out in the queue process. such as sending emails, push notifications e.t.c

Producer - The producer would represent the instantiated queue constructor. Many Jobs can be added to the specified named queue object; for processing to be carried out. such as push notifications, notifications, emails. The options argument will have options such as delay (when to execute the job from the time of creation), attempts (number of retries when a job fails), rate-limiting of jobs e.t.c.

Let's get started with a simple queue system implementation with Bull

Bull sells its queue system implementation as

The fastest, most reliable, Redis-based queue for Node. Carefully written for rock-solid stability and atomicity.

Bull is fully is compatible with typescript, well tested, queue system UI, and built on top of Redis cache with a lot of helpful abstraction methods and functionalities.

Of course, the community could use more abstracted systems to simplify solutions when using the plethora of tools with their surging complexities 😅.

Implementation

• Install Redis locally or set up a Redis docker container as below; let's add a Dockerfile and a docker-compose.yml:

# docker-compose.yaml

# pull a light-weight redis image from docker
# open a reverse port 
services:
 simple_queue_redis:
    image: redis:6.2-alpine
    container_name: simple_queue_redis
    environment:
      - ALLOW_EMPTY_PASSWORD=yes
    ports:
      - 6379:6379
    volumes:
      - ./redis:/db

 # docker-compose.yaml
FROM node:14-alpine

# Create app directory
WORKDIR /app

# Install dependencies
RUN yarn install
RUN yarn

# Copy source files from the host computer to the container
COPY . .

# Run the app
CMD ["yarn", "start:dev"]

• Install the Bull package from NPM using the following command.

npm install bull --save

• Create queues and processes. We assume that we're working in a type-safe environment, hence the use of typescript. Create a new file named notificationQueue.ts and paste the following code. Here we assume we're working on a notification job.

The NotificationQueue instance, in this case, would represent the producer and the notification job gets added to the consumer, the notificationProcess.

  // ./queues/notificationQueue.ts

import Queue from 'bull';

import { notificationProcess } from '../processes';

// producer
const NotificationQueue = new Queue('Notification', process.env.REDIS_URL);

// consumer
NotificationQueue.process(notificationProcess);

export const addNewNotification = (data: {
title: string,
body: string,
url: string,
}
) => {
  NotificationQueue.add(data);
};

export default NotificationQueue;

The following code block defines the implementation of the push notification job from the server.

We're working with pusher beams for sending push messages from the server. it's easy to use, has good documentation, and helps abstract the complexities of socketing a server to allow messaging between the server and client 😀.

Install the pusher Nodejs SDK from NPM using the following command.

npm i @pusher/push-notifications-server

We won't be focusing on how pusher works in this tutorial, the official docs is a good resource.

  // ./processes/notification.process.ts

import { Job } from 'bull';
import PushNotifications from '@pusher/push-notifications-server';

const beamsClient = new PushNotifications({
  instanceId: process.env.PUSHER_BEAMS_INSTANCE,
  secretKey: process.env.PUSHER_BEAMS_SECRET,
});

const NotificationProcess = async (job: Job) => {
  const {
    title,
    body,
    url,

  } = job.data as {
title: string,
body: string,
url: string,
};

  await beamsClient.publishToInterests([userId], {
    web: {
      notification: {
        title,
        body,
        deep_link: `${process.env.BASE_URL}${url}`,
      },
    },

  });
 return true;
}

export default NotificationProcess;

Save the file and run it using node ./index.ts. A notification will be sent to the subscribed client. (not covered in this tutorial).

// ./index.ts
import { NotificationQueueHelpers, } from '.../queues';

...

 NotificationQueueHelpers.addNewNotification({
    title: "simple queue system",
    body: "push notification sent...",
    url: `/notification/${user.id}`,
  });

Wrap-up

This is a simple example of a queue system using Bull to create queues and schedule Jobs. As mentioned above bull offers a lot more options to manage the Redis caching system behind the scenes such as Event listeners, Rate limiters, Job types, delayed jobs, number of retries, using Cron expressions, and etc.

The official documentation is still a great resource worth exploring, do check it out.

Thanks for the audience and I hope you found this article helpful 🤗. Feel free to reach out to me on Github, Twitter and LinkedIn. Do drop a like, comment, and share 😌.

Happy coding 💙!

Asynchronous programming in JavaScript can only be achieved with functions being first-class citizens of the language. Thus functions can be passed around like any other variable to other functions.

Functions that can take other functions as arguments are called higher-order functions.

In Javascript, one efficient way of handling asynchronous or expensive operations is using promises. This is because in an asynchronous operation there is no guarantee of exactly when the operation will complete and the result it will return. A promise object thus helps us to cater for the intermediate state of the operation, whether it is successful(resolve), pending(ongoing), or fails(reject).

Promise.resolve() //success
Promise.reject() // fail

Why Promises?

We established above that a Promise object represents the result of an asynchronous function. Promises can be used to avoid chaining of callbacks, referred to as the Callback hell.

let's get started by converting Node-styled callback to promise.

Assumptions, let's set the tone for the discussion 😃

• You're familiar with javascript
• And you have some experience with the Node environment; if this is all checked ✅, we're set to learn about promises 😇.

It's easier to work with promises when working on async operations, but unfortunately, Nodejs has many of its APIs written with callbacks. (Promises were updates to the ECMAScript)

Here's an example using the file stream API. fs.readFile

const fs = require('fs') 

fs.readFile(filePath, options, callback)

Here, the callback function contains at least two arguments. The first argument must be an error object and the data returned.

fs.readFile('file-path', (err, data) => {
  if (err) {
     //throw new error
  } 
    //do something with data return

})

Below we convert a callback styled Node API to a promise, using the util.promisify API

const util = require('util');
const fs = require('fs');

const readFilePromise = util.promisify(fs.readFile)
readFilePromise('file-path').then((data) => {
    //do something with data return
}).catch((error) => {
    //throw new error
});

We can also use the async/await operation to resolve a promise.

const readFilePromise = util.promisify(fs.readFile)

async function(){
  const data = await readFilePromise;
  console.log(data);
}

Let's Create Promises

In javascript, we can create asynchronous functions that return a promise 😊 . see the example below.

   const myPromise = new Promise((resolve, reject) = {
     setTimeout(() => 
       {
             console.log('resolving the promise ...');
             resolve('resolved!');
       }, 5000); // usually would be an asychronous operation

    reject('rejected!');
});

console.log(myPromise);

We can chain the Promises using .then() and .catch() higher-order functions. The .then() accepts a callback as an argument and invokes the resolved Promise. The .catch() accepts a callback and is executed when the Promise is rejected.

see the example below.

const resolvedCallback = (data) => console.log(data);
const rejectedCallback = (err) => console.log(err);

myPromise.then(resolvedCallback).catch(rejectedCallback);

More examples handling Multiple independent promises

Unlike the examples we wrote above, we can return the state of a promise object without conditions. this might be useful when we need to resolve or reject multiple independent asynchronous operations. it's as simple as writing;

Promise.resolve();
Promise.reject();

We notice in the Node environment, the Promise object is globally available, interesting right! 😎

When dealing multiple independent promises, the methods Promise.all() and Promise.race() can be useful. Both methods accept an array of promises as an argument.

While using Promise. all(), the .then method is only invoked after the last promise is resolved.

const promiseArr = [
  Promise.resolve("data1")
  Promise.resolve("data2")
  Promise.resolve("data3")
]

Promise.all(promiseArr).then(results =>
  console.log(results)
)

// [ data1, data2, data3 ]

While using Promise.race(), the .then method is invoked only when the first promise is resolved.

Promise.race(promiseArr).then(results =>
  console.log(results)
)

// data1

One thing to note as well is that the least expensive function will always resolve first. This enables us to run multiple asynchronous operations and act upon the one that finishes first.

Wrap-up ✍️

• A promise represents the state of execution of a function
• A promise has 3 states; resolve, reject and pending.
• Using async and await is a sugar coating wrapped around promises and it makes working with promises easy and readable
• Await ensures it resolves the promise being executed and encourages return error early.
• Promises and async/await accomplish the same thing.
  • They make retrieving and handling expensive functions/asynchronous data easier.
  • They eliminate the need for callback hell/nested functions
  • Simplify error handling.
  • Reduce extraneous code
  • Make waiting for multiple concurrent calls to return and adding additional code in between calls easier.

Thanks for the audience and I hope you found this article helpful getting started with promises in Javascript 🤗. Feel free to reach out to me on Github, Twitter and LinkedIn. Do drop a like, comment, and share 😌.

Find out more about Graceful asynchronous programming with Promises

What are ORMs?

Object-relational mapping conceptualizes the idea of being able to write queries like the one below, as well as much more complicated ones, using the object-oriented paradigm of your preferred programming language. Essentially, ORMs help abstract the complexities of writing custom SQL queries to a database.

SELECT * FROM user where countryId = 2

Well, just like any other tool or library, we have pros and cons to using them. For the most part, I stumbled on Prisma, and it's arguably an excellent tool for low to medium-level complexity and enterprise apps.

• You easily scaffold a Nodejs server with a database quickly with the least amount of code.
• It's open-source, it's compatible with a number of SQL such as PostgreSQL, MySQL, and SQLite.
• Out of the box support for typescript.
• Many tools aren't properly documented and we have to rely on the community for help when stuck, but Prisma documentation is quite delightful and very detailed.

Next up we will look at some prerequisites for this tutorial, featuring Prisma, Express.js + typescript, and PostgreSQL.

Prerequisites

This tutorial assumes you've had some experience with Nodejs ;

• You have the Nodejs LTS installed
• Scaffold an Expressjs app
• Basic knowledge of Javascript/typescript
• Some idea of how ORMs interact with databases

Implementation

Lets set Up the Express.js app

• We’ll begin by setting up the Express.js app for a Movies listing API demo project. Navigate into a new folder First, then run the following to create a new project:

we also assume you're using a Unix shell/terminal

mkdir movies-listing-api
cd movies-listing-api
npm init -y

• Next, install Express.js, our minimalistic framework for the Star Wars REST API:

npm install --save express

• Next, setup typescript dev dependencies

npm install typescript --save-dev
npm install @types/node --save-dev

• Next, Create a tsconfig.json file The tsconfig.json is where we define the TypeScript compiler options. We can create a tsconfig with several options sets. it should be at the root directory of the project.

npx tsc --init --rootDir src --outDir build \
--esModuleInterop --resolveJsonModule --lib es6 \
--module commonjs --allowJs true --noImplicitAny true

At this point, you should have a tsconfig.json that looks like this:

{
  "compilerOptions": {
    /* Basic Options */
    // "incremental": true,                   /* Enable incremental compilation */
    "target": "es5",                          /* Specify ECMAScript target version: 'ES3' (default), 'ES5', 'ES2015', 'ES2016', 'ES2017', 'ES2018', 'ES2019' or 'ESNEXT'. */
    "module": "commonjs",                     /* Specify module code generation: 'none', 'commonjs', 'amd', 'system', 'umd', 'es2015', or 'ESNext'. */
    "lib": ["es6"],                     /* Specify library files to be included in the compilation. */
    "allowJs": true,                          /* Allow javascript files to be compiled. */
    // "checkJs": true,                       /* Report errors in .js files. */
    // "jsx": "preserve",                     /* Specify JSX code generation: 'preserve', 'react-native', or 'react'. */
    // "declaration": true,                   /* Generates corresponding '.d.ts' file. */
    // "declarationMap": true,                /* Generates a sourcemap for each corresponding '.d.ts' file. */
    // "sourceMap": true,                     /* Generates corresponding '.map' file. */
    // "outFile": "./",                       /* Concatenate and emit output to single file. */
    "outDir": "build",                          /* Redirect output structure to the directory. */
    "rootDir": "src",                         /* Specify the root directory of input files. Use to control the output directory structure with --outDir. */
    // "composite": true,                     /* Enable project compilation */
    // "tsBuildInfoFile": "./",               /* Specify file to store incremental compilation information */
    // "removeComments": true,                /* Do not emit comments to output. */
    // "noEmit": true,                        /* Do not emit outputs. */
    // "importHelpers": true,                 /* Import emit helpers from 'tslib'. */
    // "downlevelIteration": true,            /* Provide full support for iterables in 'for-of', spread, and destructuring when targeting 'ES5' or 'ES3'. */
    // "isolatedModules": true,               /* Transpile each file as a separate module (similar to 'ts.transpileModule'). */

    /* Strict Type-Checking Options */
    "strict": true,                           /* Enable all strict type-checking options. */
    "noImplicitAny": true,                    /* Raise error on expressions and declarations with an implied 'any' type. */
    // "strictNullChecks": true,              /* Enable strict null checks. */
    // "strictFunctionTypes": true,           /* Enable strict checking of function types. */
    // "strictBindCallApply": true,           /* Enable strict 'bind', 'call', and 'apply' methods on functions. */
    // "strictPropertyInitialization": true,  /* Enable strict checking of property initialization in classes. */
    // "noImplicitThis": true,                /* Raise error on 'this' expressions with an implied 'any' type. */
    // "alwaysStrict": true,                  /* Parse in strict mode and emit "use strict" for each source file. */

    /* Additional Checks */
    // "noUnusedLocals": true,                /* Report errors on unused locals. */
    // "noUnusedParameters": true,            /* Report errors on unused parameters. */
    // "noImplicitReturns": true,             /* Report error when not all code paths in function return a value. */
    // "noFallthroughCasesInSwitch": true,    /* Report errors for fallthrough cases in switch statement. */

    /* Module Resolution Options */
    // "moduleResolution": "node",            /* Specify module resolution strategy: 'node' (Node.js) or 'classic' (TypeScript pre-1.6). */
    // "baseUrl": "./",                       /* Base directory to resolve non-absolute module names. */
    // "paths": {},                           /* A series of entries which re-map imports to lookup locations relative to the 'baseUrl'. */
    // "rootDirs": [],                        /* List of root folders whose combined content represents the structure of the project at runtime. */
    // "typeRoots": [],                       /* List of folders to include type definitions from. */
    // "types": [],                           /* Type declaration files to be included in compilation. */
    // "allowSyntheticDefaultImports": true,  /* Allow default imports from modules with no default export. This does not affect code emit, just typechecking. */
    "esModuleInterop": true,                  /* Enables emit interoperability between CommonJS and ES Modules via creation of namespace objects for all imports. Implies 'allowSyntheticDefaultImports'. */
    // "preserveSymlinks": true,              /* Do not resolve the real path of symlinks. */
    // "allowUmdGlobalAccess": true,          /* Allow accessing UMD globals from modules. */

    /* Source Map Options */
    // "sourceRoot": "",                      /* Specify the location where debugger should locate TypeScript files instead of source locations. */
    // "mapRoot": "",                         /* Specify the location where debugger should locate map files instead of generated locations. */
    // "inlineSourceMap": true,               /* Emit a single file with source maps instead of having a separate file. */
    // "inlineSources": true,                 /* Emit the source alongside the sourcemaps within a single file; requires '--inlineSourceMap' or '--sourceMap' to be set. */

    /* Experimental Options */
    // "experimentalDecorators": true,        /* Enables experimental support for ES7 decorators. */
    // "emitDecoratorMetadata": true,         /* Enables experimental support for emitting type metadata for decorators. */

    /* Advanced Options */
    "resolveJsonModule": true                 /* Include modules imported with '.json' extension */
  }
}

• Next, set up the entry point of the application by checking out into the project/src directory and create two files app.ts and server.ts .

mkdir &&  cd src
touch server.ts
touch app.ts

the files should look like this : This will contain the express routing and handle API requests.

#app.ts
import express, { Application } from 'express';

// express app
const app: Application = express();


export default app;

This Nodejs server, serves the express application and exposes it over process.env.PORT or 8000;

#server.ts
import app from './app';

// constants
const PORT = process.env.PORT || 8000;

// listen
app.listen(PORT, () => console.log(`⚡ on http://localhost:${PORT}/api`));

• Next up, we’ll set up a PostgreSQL database to create our tables for the star-wars movie API.

Set Up a Local PostgreSQL Database with docker

Create a Postgres image at the root of the project directory, build the image, and run it.

touch postgres.yaml

the docker-compose file should look this:

version: '3'
services:

  postgres:
    # For more details on configuring the Postgres Docker image, see:
    #   https://hub.docker.com/_/postgres/
    image: postgres:10.3-alpine #light weight 

    # Expose the default Postgres port on localhost
    ports:
      - '5432:5432'
      # allow connections from the host machine
      # an instance of the postgres image = postgres_docker(container)
    network_mode: bridge # port available on localhost 
    container_name: postgres_docker

    environment:
      POSTGRES_USER: 'postgres'
      POSTGRES_PASSWORD: 'pgpass'
      POSTGRES_DB: 'postgres'

    # Copy files from dbinit into the image so that they will be run on boot
    # mount db
    volumes:
      - ./initdb:/docker-entrypoint-initdb.d

If you want to run a PostgreSQL database with Docker, use the following command:

docker compose -f postgres.yaml up

The docker run command creates a new docker container named postgres_docker, exposing the container port 5432 to the local port 5432.

Our database is up and running. Next up, we will add Prisma to our Node.js project and create our schema.

Install Prisma ORM to Your Node.js Project

Run the following command, which installs Prisma as a dev dependency

npm install prisma --save-dev

After that, initialize the Prisma schema with the following command:

npx prisma init

Next, add the database connection string to the env file, it should look this:

#.env
DATABASE_URL="postgresql://postgres:pgpass@localhost:5432/movies_storage?schema=public"

Add Models and Run Prisma Migration

the Prisma folder at the root directory contains prisma.schema file, in it we define our database tables for Movies and Users with their relation.

datasource db {
  provider = "postgresql"
  url      = env("DATABASE_URL")
}

generator client {
  provider = "prisma-client-js"
}

model Movie {
  id        Int      @id @default(autoincrement())
  title   String @unique
  description     String

  createdAt DateTime @default(now())
}

We have added One table to the schema. The Movies table, which has an auto-increment ID, the title field that must be unique, description, and createdAt field that defaults to the current date.

To convert these models into PostgreSQL database tables, run the following command:

prisma migrate dev --name init --preview-feature

This generates the migration file, that contains the SQL that creates the tables in the PostgreSQL database.

API to interact with movie listings

We will Update the app.ts file we created in a previous step and add the POST Movies and GET Movies API.

const app: Application = express();

import { PrismaClient } from '@prisma/client';
const prisma = new PrismaClient();

app.get('/', (req, res) => {
  res.json({message: 'alive'});
});

app.post('/movies', async (req, res) => {
  const {title, description} from 'req.body';
await prisma.movie.create({
     data: {
     title,
     description,
}
    });
   res.json({
    message: "movie created"
  });
});

app.get('/movies:id', async (req, res) => {
const movie =  await prisma.movie.findUnique({
   where:{
   id: req.params.id
},
  });

  res.json({
    data: movie,
  });
}

app.get('/movies', async (req, res) => {

  const allMovies =  await prisma.movie.findMany({});

  res.json({
    data: allMovies,
  });
});

export default app;

Wrap-up

We built a Movie listing API with Express.js+typescript and Prisma ORM, using a PostgreSQL database running on Docker. I know we barely scratched the surface of what Prisma offers, nonetheless, I'm hoping this gives you some insights into adding Prisma ORM to your Nodejs apps.

To finish up, I suggest you further explore some great Prisma ORM features. For instance, query optimization, database transactions, and aggregate queries are good reads.

Thanks for the audience and I hope you found this article helpful 🤗. Feel free to reach out to me on Github, Twitter and LinkedIn. Do drop a like, comment, and share 😌.

Happy coding!

For me, coding brings some sense of satisfaction. The skill of being able to program computers to do what you want, learning how applications work under the hood and I loved video games... yea for some techies, that was the spark.. it was exciting! I had a useful skill! 🤩. And The curiosity has kept me relentlessly getting better 😃.

Asides from freelancing and teaching coding, I had my first coding job at a hospitality firm in 2019 and since then I've had the resolve, " Coding every day can be fun until it isn't 😂! ".

During these years I have faced the good, the bad, and the ugly side of a programming job 😅.

I'll talk about tips I wish I knew when I started out learning tech 🖊️.

Programming is not about coding, programming is about solving problems with coding.

In my opinion, this mindset can change a lot about your perspective and accelerate your growth.

Being able to see programming not as a hobbyist but as a professional who solves problems with technology. if we adopt this mentality the developer community will be less toxic...leave out emotions and identify opportunities as a software developer.

The truth is, software development has a lot of potentials, not only as a career but as a tool for entrepreneurship. As a developer, you have the ability to create anything that comes to mind with only the help of a computer. The skill to code anything by learning programming languages. The possibility to turn ideas into reality.

Communication Skills Are Just As Important As Coding Skills.

When we start out learning to code we invest a lot in our growth. for me, I used up all my free time coding and coding and more coding 😆!

Well while coding is a great skill to have... it's not enough 🤔!

Being able to sell yourself as a good developer is as important as being a good developer. Don’t be afraid to speak your mind and don’t wait for someone to offer you a senior / tech-lead position. Be this person, first 😉!

In the tech space it's important you sell yourself as a professional and not a hobbyist building and maintaining an online presence. Twitter, networking events, writing/reading articles and blogs, and communicating effectively are places to start.

hack4justice2020 hackathon

Tech is communal. that's why we have teams, pairing, open-source, stack overflow, conferences, books, hackathons, and blogs. Take advantage of this! 🚀

Be Open to change and willing to learn.

Software engineering has changed a lot 🤔. No one reinvents the wheel (waste of time). Time is a scarce commodity to software engineers 😏 Professionals have been reduced to gluing technologies together, from third-party libraries, APIs, frameworks, and many times Open-source software, with the aim of producing and innovating a product(SAAS).

Staying relevant is a constant need for all professionals, but perhaps even more so for Developers. With technology changing at such a fast clip, it will be survival of the fastest in many parts of the industry.

• Stay current with industry news/ tech stack and OTHER news

• I know we love to build. We love to code 😅...Just Use modern libraries. They'll generally make your life easier. And At least companies use them 🤷‍♂

• Don't make the big mistake to stop your education. You’ve worked hard to get where you are, and much of that doubtlessly included education (whether formal or informal).

• The best way to learn is to connect with your peers; online communities are a great place to do this. Twitter has a great dev community. Take advantage of it and learn.

• Attend meetups and tech conferences

• Simply reading new job descriptions and staying connected to the market will tell you a world of information about what practical skills are generating the most interest and how to level up.

  Well, take advantage of the changing modern sphere of software.

You will have imposter syndrome.

Moments you feel like a fraud running out of luck and soon to be figured out 😩. It's normal and it happens to everyone and at whatever level you are. When we feel this way it's important we try not to make it deter us or we dwell so much on how much we suck at programming 😏.

How to counter our internal critics

• Accept and move forward: You won't know everything. it's okay to know a few things.
• Share your knowledge
• Try to itemize your accomplishments
• Give kudos to awesome people and help people
• Teach what you know
• What gives you credibility?
  • knowledge
  • Experience
  • Contributions

• Write a bio, what to include:

  • Roles: e.g team lead, project manager, volunteer, backend developer, frontend developer, community lead, mentor.
  • Skills: e.g show empathy, resolve conflicts, team player, communicate effectively.
  • Strengths: I'm relentless, I strive to be better.
  • Experiences: I've contributed value to employers, I have skills I'm proud of.
  • Values: positive attitude, communication, open-mindedness

  The tips outlined above are worth trying out, For a couple of reasons see it as a good thing, you have nothing to lose 🤷‍♂️. it could help you keep learning and growing. it could serve as a moral compass helping you keep the right balance between humility and confidence 😀.

Take care of your physical and mental health.

There's a thing as being addicted to programming. Everything has a cost. Let's always keep the cost In check.

find a balance between work and life. in the long run, we're not robots. we can get overwhelmed by work, get tired, lose motivation, relationships can suffer and we can burn out.

In the long run, It's not a healthy habit that we work so much that we skip meals, rest, and some leisure time we don't stress overwork. Programming can be both mentally and physically exhausting. it's important we strive to rest well, we'll be surprised by how much proper night rest, can lead to a productivity boost at work 😃.

You won't know everything.

If it's one thing I've learned as a programmer: it's to approach programming languages as just tools to get things done. The things that matter most are on a level above language syntax or libraries.

Don't get caught up in needless debates on which is best. As a matter of fact, the best skill a developer can have is being a problem solver.

You're always going to be at a point where you're going to need to know more and that's always going to be the case, especially in this industry. so you're never going to have all your docs in a row, you're gonna need to study more. so just start building something. anyone who's willing to hire you just wants to see someone that's willing to learn wants to pick up new things and gets the job done.

Take your time and learn the fundamentals well.

It's okay to have big goals and plans. But it could sometimes be a snare to rush and skip the essential process of learning. Knowing and tracking what you know is important.

My advice to a budding developer would be to pick one language and learn it well. don't be swayed by the hype in the community, many times it's influenced by people's sentiments and not the market need. for example, JavaScript is not perfect but it works. so learn it well.

When it comes to software development, think long-term. Software Development is a marathon, not a sprint.

You can't learn 10 technologies in a week and build a full-scale app the second week 💔.

Take your time, and enjoy the journey. There is no rush. 😊

I am not where I want to be yet, and every day is a learning experience for me! 💪

Here we go! that concludes my top 7 tips article on this subject. I hope this helps a newbie do better and make better decisions.

Thanks for the audience and I hope you found this article helpful 🤗. Feel free to reach out to me on Github, Twitter and LinkedIn. Do drop a like, comment, and share 😌.

CASL.JS at its core is an Isomorphic Authorization JavaScript library ". 🤔

The fancy word “isomorphic” means that you can use the library on both the frontend and backend in exactly the same way. - Sergii Stotskyi

CASL (pronounced /ˈkæsəl/, like a castle) is an isomorphic authorization JavaScript library that restricts what resources a given client is allowed to access. It's designed to be incrementally adoptable and can easily scale between a simple claim-based and fully featured subject and attribute-based authorization. It makes it easy to manage and share permissions across UI components, API services, and database queries.

CASL is versatile, you can start with simple role-based access control and scale your apps to include fully-featured attribute-based access control.

CASL is declarative, it allows you to define permissions in the memory of the server-side using a domain-specific language that matches your business requirements almost word for word.

CASL is TypeSafe, it’s written in TypeScript, this makes apps safer and the developer experience more enjoyable.

CASL is small, it’s just ~4.5KB mingzipped and can be even smaller, thanks to tree-shaking! The minimum size is ~1.5KB.

Backstory

So while working on a Nodejs application at work; the team spent weeks researching and working on a hybrid access control framework that will be both scalable and dynamically change as the business needs require it.

Recent developments have shown that attribute-based access control (ABAC) can provide flexible and fine-grained access control in both dynamic distributed systems and enterprise applications. As only the attributes of the subject, object, and environment are considered, most current solutions of typical Role based-access-control (RBAC) schemes cannot scale permissions to resources for a range of users with varied attributes.

Also in an (ABAC) framework, the objective attributes are gained or defined by the resources themselves under certain conditions, such as time, location, IP address, and E.T.C.

So in the hybrid access control framework, permission to resources is modeled across roles, claims, attributes of users on the platform; such as user-defined attributes specified by the profile owner, such as gender, name, job, hometown, hobbies, and etc.

Well building a custom hybrid access control framework, as versatile as discussed above can be quite resource-heavy, especially for small teams 🙄.

We found CASL JS authorization library to be awesome, easy to adopt, heavily maintained, and efficiently capable of scaling with the business for dynamically managing permission to resources 😃.

ASSUMPTIONS. I assume you already have some knowledge of Nodejs and managing permissions in your apps by claims or roles 😌.

Get started - TALK IS CHEAP 😆

1). Install @casl/ability as a dependency in your Nodejs application:

npm i @casl/ability

2). Define Abilities

There are 3 ways you can define abilities:

- using defineAbility function
- using AbilityBuilder class
- using JSON objects

In this example, we'll be using the defineAbility function

This function allows the creation of an Ability instance using can and cannot methods. It allows to define and use Ability instance without writing too much code.

//ability/defineAbility.js


const { AbilityBuilder, Ability } = require("@casl/ability");

/**
 * @param user contains details about logged in user: its id, name, email, etc
 */
// Define abilities for subjects here

// permissions on Organization
exports.defineAbilitiesOnOrganizationFor = (user) => {
  const { can, cannot, rules } = new AbilityBuilder(Ability);

  // condition == True
  if (user.isAdmin) {
    // can manage (i.e., do anything) own Oganization
    can(
      "manage", // can do everything
      "Oganization", //  Organization collection
      ["email",
        "phone",
        "password",
        "firstName", // feilds that can be managed
        "lastName"
      ],
      { _id: user._id } // condition , if OrgId=user.Id (belongs to an Org)
    );

    // But cannot delete Oganization
    cannot(
      "delete", // cannot delete Organization any Organization
      "Organization" // collection Organization
    );
  }

  if (user.isSuperAdmin) {
    // define the abilities for superAdmin on organization(subject)
    can(
      "manage",
      "Organization"
    );
  }

  return new Ability(rules);
};

in the above implementation we defined permissions for an "admin" and "super admin", we further set conditions for the organization as a resource that can be managed by both users.

CASL operates on the abilities level, that is what a user can actually do in the application. An ability itself depends on the 4 parameters (the last 3 are optional):

User Action Describes what users can actually do in the app. User action is a word (usually a verb) that depends on the business logic (e.g., prolong, read). Very often it will be a list of words from CRUD - create, read, update and delete.

Subject The subject or subject type on which you want to check user action. Usually, this is a business (or domain) entity (e.g., Subscription, Article, User, Organization). The relation between subject and subject type is the same as the relation between an object instance and its class.

Fields Can be used to restrict user action only to matched subject's fields (e.g., to allow the admin to update a field of a resource and disallow to update certain fields)

Conditions Criteria that restricts user action only to matched subjects. This is useful when you need to give permission on specific subjects (e.g., to allow the user to manage their own profile accounts)

YES; easily business requirements can be translated into CASL's rules 🤩.

3). Database integration CASL has a complementary package [@casl/mongoose] which provides easy integration with MongoDB and [mongoose].

To install dependency: npm install @casl/mongoose

// Organization.js


import { AbilityBuilder } from '@casl/ability';
import { accessibleRecordsPlugin } from '@casl/mongoose';
import mongoose from 'mongoose';
import {defineAbilitiesOnOrganizationFor} from 'ability/defineAbility.js'

mongoose.plugin(accessibleRecordsPlugin);

const user = getUserLoggedInUser(); // app specific function

const ability = defineAbilitiesOnOrganizationFor(user);


const Organization = mongoose.model('Organization', mongoose.Schema({
  email: String,
  isAdmin: Boolean, 
  phone: Number,
  password: String,
  content: String,
  createdAt: Date,
  firstName: String,
  lastName: String

}))

Organization.plugin(accessibleRecordsPlugin)

module.exports = mongoose.model('Organization', Organization)

4). Check abilities

// ../controllers/organizationController.js

const { NotFound, Unauthorized, InternalServerError } = require("http-errors");
const Organizations = require("../models/organization");

async function permissionChecker(model, defineAbilitiesOnSubjectFor, userId) {
  try {

   // get user and all attributes
    const user = await model.findOne({ _id: userId });
    if (!user) {
      throw new NotFound("user not found");
    }
    // get ability
    const ability = defineAbilitiesOnSubjectFor(user);
     return ability;

  } catch (error) {
    throw new Unauthorized(error.message);
  }
}

let action = "update" // specify the type of action on the resource, to check for
let subject = "Organization"  // resource that needs permission and access control

// check for permission using the permission checker
    const ability = await permissionChecker(
      Organization,
      defineAbilitiesOnOrganizationFor,
      user
    );

if(ability) {
   try {
        const organization = new Organization();
        organization.set(data); // app specific data
        ForbiddenError.from(ability).throwUnlessCan(action, organization);
        await organization.save();
   }catch(error){
      throw new Unauthorized(error.message);
   }
} else{
   throw new InternalServerError(error.message);
}

Yea, it's that simple to implement a scaleable in memory permission management and resource authorization with CASL in your Nodejs apps 😊.

CASL is progressively adaptable, which means you can start your project with the simple role-based authorization and evolve it later when your app functionality evolves.

Thanks for the audience and I hope you found this article helpful 🤗. Feel free to reach out to me on Github, Twitter and LinkedIn. Do drop a like, comment, and share 😌.

FIND OUT MORE

• CASL JS Documentation

Many RBAC (Role-Based Access Control) implementations differ, but the basics are widely adopted since it simulates real-life role (job) assignments. - Onur Yıldırım.

Role-based access control (RBAC) is a method of modifying access and privileges based on the roles of individual users or user groups within an organization.

RBAC lets members of an organization have access, to only the resources they need to do their jobs and restricts or prevents unlimited privileges to resources that don't pertain to them.

RBAC vs ABAC

Role-based access control and attribute-based access control (ABAC) are both types of access control methods, but their approaches are different. A use case that necessitates an RBAC system, would be if we intend to grant privileges on the application depending on the roles of users.

While ABAC grants access based on a combination of attributes, i.e., user attributes, resource attributes, attributes associated with the system or application to be accessed, environmental attributes, and so on. For example, User attributes may include biodata information, uniquely identifies, role, security clearance, and etc.

Back Story

On a recent project, my team was tasked with implementing a custom Role-based access control framework for MongoDB(NoSQL DB), using the mongoose ORM schema. In this access control framework we'll be considering 3 entities :

1. Roles
2. User
3. Resources

Implementation context

• The User has a Role
• The Application has defined Resources and,
• The role has privileges & permissions to Resources

let's write some mongoose schema's for our collections 😄

via GIPHY

Schema

resources.models.js

/* RBAC - role-based access control
 first create roles; role =[ "user", "guest", "organization",  "superadmin" "globalsuperadmin"]
 next, create resouuce, add their roles_id and permmissions
 finally create users and indcate thier roles
*/

    const mongooseClient = app.get('mongooseClient');
    const { Schema } = mongooseClient;

    const resources = new Schema({
      name: { type: String, required: true },
      slug: { type: String, required: true },
      resources_roles: [{
        roles_id: { type: Schema.Types.ObjectId, ref : 'Roles' },
        roles_name: { type:  String },
        create: { type: Boolean },
        delete: { type: Boolean },
        update: { type: Boolean },
        read: { type: Boolean },
      }]
    }, {
      timestamps: true
    });

    return mongooseClient.model('resources', resources);

roles.models.js

   const mongooseClient = app.get('mongooseClient');
    const { Schema } = mongooseClient;

  module.exports = function (app) {
    const mongooseClient = app.get('mongooseClient');
    const { Schema } = mongooseClient;
    const roles = new Schema({
      name: { type: String, required: true },
      slug: { type: String, required: true },
    }, {
      timestamps: true
    });

    return mongooseClient.model('roles', roles);
  };

users.models.js

 const mongooseClient = app.get('mongooseClient');
    const { Schema } = mongooseClient;


  module.exports = function (app) {
    const mongooseClient = app.get('mongooseClient');
    const { Schema } = mongooseClient
    const users = new mongooseClient.Schema({
      email: {type: String, unique: true, lowercase: true},
      password: { type: String },
      first_name: { type: String },
      last_name: { type: String },
      roles: { type: Schema.Types.ObjectId, ref : 'Roles' },

    }, {
      timestamps: true
    });

    return mongooseClient.model('users', users);
  };

Now when you want the permissions of a role on a resource, you just look up role_id and resource_id and check for which permissions are set to true in the resources collection.

Authorization Middleware

users.post('/', getAuth, someMethod)

Assuming you have a token of some sort on the request that identifies the user making the post, and attaching the user instance to the request object you can do this:

// pseudo code
getAuth = function (req, res, next) {
  if(req.user) { 

// query to get the user role's  permissions for a resource
    if(token){
    // handle jwt token authenticity and decrypt payload 

   // get permission handler
    db.getPerms({ role_id: req.user.role_id, resource_id: req.resource.id})
    .then((perms) => {
       var allow = false;
       // mapping of methods to permissions
       perms.forEach(function(perm){
           if (req.method == "POST" && perms.create) allow = true;
           else if (req.method == "GET" && perms.read) allow = true;
           else if (req.method == "PUT" && perms.write) allow = true;
           else if (req.method == "DELETE" && perm.delete) allow = true;

       })
       if (allow) next();
       else {
            res.status(403).send({error: 'access denied'});
        }
    })
    .catch((err)=> {
       //handle your reject and catch here
    })
} else{
    res.status(400).send({error: 'invalid token'})
}

This is some Pseudo-code, showing how the authentication middleware can be written.

This isn't the most optimized framework for implementing RBAC, so I'm looking forward to your feedback 😄!!

Thanks for the audience and I hope you found this article helpful 🤗. Feel free to reach out to me on Github, Twitter and LinkedIn. Do drop a like, comment, and share 😌.

For example, a web server such as Apache handles requests made from the web clients and serves up static or dynamic content or HTML web pages as a response. Each request is handled like a small program, in a series of response and request cycles.

How ideal it'll be for large programs to run their smaller components independently and simultaneously 🤔 !?

Making progress on more than one task simultaneously is known as concurrency.

Golang, At the language level offers some great concurrency primitives and makes it extremely easy to implement a concurrent system. Go has rich support for concurrency using

• Goroutines &
• Channels.

Goroutines

Goroutine always works in the background. They are functions executing concurrently with other goroutines in the same thread or set of threads. To create a goroutine we use the keyword go followed by a function invocation:

package main

import (
    "time"
    "fmt"
)

func say(s string){
    for i:=0; i < 3; i++ {
        fmt.Println(s)
        time.Sleep(time.Microsecond*100)
    }

} 
func main() {
    start := time.Now()

    go say("Hey")
    go say("There")

    elapsed := time.Since(start)
    fmt.Println(elapsed)

//the program runs concurrently
}

Every program contains at least one Goroutine and that Goroutine is known as the main Goroutine.

This program above consists of three goroutines. The first goroutine is implicit, it is the main function itself and all the goroutines are working under the main Goroutines .

The second and third goroutine is created when we invoke a function; In this case by calling go say("Hey") & go say("There").

By convention, when a function is invoked our program will execute all the statements within the function and then return to the next line following the invocation. But with a goroutine, we return immediately to the next line and don't wait for the function to complete.

We can investigate this concurrency and the program time of execution by setting some delay.

A hack 👨‍💻 I found was to use the sleep method time.Sleep(time.Second) .

Without this, the program would exit before being given the opportunity to print all the strings ''hey'' and "there".

Goroutines are lightweight and we can easily create many of them in our programs. We can modify the above program to run 20 😃goroutines by doing this:

func main() {

start := time.Now()

  for i := 0; i < 10; i++ {
    go say("Hey")
    go say("There")

  }

time.Sleep(time.Second)
elapsed := time.Since(start)
fmt.Println(elapsed)

}

Channels

Channels are used for communication and sometimes synchronization between two executing goroutines. Here is a sample program using channels:

package main

import (
    "fmt"
    "time"
)

func main() {
    // Channels are defined using the make function,
    // VariableName:= make(chan value-type)

    ch := make(chan string)

    // this sends data to the channel
    go greeting("Hello", ch)
    for {
        // We can indicate the direction of the data flow by using an arrow ch <- data

              msg := <- ch
        // and v:= <- ch receives the data from the channel!

        fmt.Println(msg)
    }
}

func greeting(s string, ch chan string) {
    for i := 0; i <= 5; i++ {
        //WE SEND A VALUE THROUGH A CHANNEL
        ch <- s
        time.Sleep(100 * time.Millisecond)
    }
    close(ch)
}

The channel is created using the makefunction and the chan keyword, it is then passed on to the call to the greeting function which accepts a string and a channel of data-type string. The for loop is created in the main channel to receive and print out every incoming message.

The close() function indicates Closing a channel; thus no more values will be sent on it. This can be useful to communicate completion to the channel’s receivers.

So using a channel like this synchronizes the two goroutines.

Conclusion

I briefly talked about the essential concepts in Golang such as goroutines and channels, which are the implementations of concurrency in Go. I hope this article helps you learn more about Golang features 😃.

I enjoyed writing this article and I hope you found this article helpful 🤗. Feel free to reach out to me on Github, Twitter and LinkedIn. Do drop a like, comment, and share 😌.

Do watch the Go-Routines video by Rob Pike and have fun coding in Golang.

Until next time…

Further Reading

• Learn More about Go concurrency

According to Docker docs, Docker is a tool that provides OS-level virtualization to deliver software in packages called containers.

Docker made application containers easy to use by providing a simplified, opinionated wrapper around existing Linux distribution invoked as containers, process control, security, and resource management technologies.

Docker excels in these two key areas:

• Packaging applications
• Process isolation and management

Docker helps you package and run applications inside isolated little boxes with all platform and application dependencies provided, thus keeping the hosting computer nice and tidy.

Docker solves problems associated with :

• Missing or incorrect application dependencies such as libraries, interpreters, code/binaries, users, Os packages, etc
• Conflicts between programs running on the same host machine such as library dependencies or ports; Example: local Django trying to use port 80
• Insufficient system resources required to run an application such as CPU and memory.

Let’s see how Docker delivers on its promise to “Build, Ship, and Run” applications easily 😃.

Assumptions: We'll be working with a Django application, Some experience working with a Django application might be needed and you have Docker & Docker Compose installed.

Packaging an Application Into An Image

This OS-level virtualization provided by Docker helps us mirror the application and platform into a docker file. The docker file should be placed at the root of the Django project.

# The first instruction is what image we want to base our container on
# We Use an official Python runtime as a parent image
FROM python:3.8.5 as production

# The environment variable ensures that the python output is set straight
# to the terminal without buffering it first
ENV PYTHONUNBUFFERED 1

# create a root directory for our project in the container
RUN mkdir /app

# Set the working directory to /app
WORKDIR /app

# Copy the current directory contents into the container at /app
COPY . /app/

# Install any needed packages specified in requirements.txt
RUN pip install -r requirements.txt

# tell the port number the container should expose
EXPOSE 8000

Docker images also allow you to define metadata and define more options to help operators run the application according to your application needs.

Setting Up A Docker-compose.yml File

For clarity, Docker Compose solves the problem of running multi-container applications at once. You thus can set the desired amount of containers counts, their builds, services, and volumes, and then with a single set of commands, you can build, run, and configure all the containers.

According to Docker-compose Docs Using Compose is basically a three-step process:

• Define your app’s environment with a Dockerfile so it can be reproduced anywhere.

• Define the services that make up your app in docker-compose.yml so they can be run together in an isolated environment.

• Run docker-compose up and Compose starts and runs your entire app.

it is also located in the root directory of the Django application. The Docker environment variable file .env is necessary when you're creating complex containers or Database service for deployments.

So let's say you have your .env file (that's the full name of the file, by the way) located in the same directory that houses your docker-compose.yml file. In the .env file, you have on these lines:

DATABASE_NAME=postgresdb
DATABASE_USER=postgresuser
DATABASE_PASSWORD=postgresuser
DATABASE_HOST=db
DATABASE_ENGINE=django.db.backends.postgresql
DATABASE_PORT=5432

A simple docker-compose.yml looks like this:

version: '3.8'

# This file defines two services: The db service and the web service.
services:
  db:
    image: postgres
    environment:
# key-value pair for the environmental variable
      - environment:
        - "POSTGRES_USER:${DATABASE_USER}"
        - "POSTGRES_DB:${DATABASE_NAME}"
        - "POSTGRES_PASSWORD:${DATABASE_PASSWORD}"
        - "POSTGRES_HOST:${DATABASE_HOST}"
    volumes:
# no data is lost  when we stop running the container, only for development
      - ./pgdata:/var/lib/postgresql/data
    ports:
# exposing port 5432 to the host machine
      - "5432:5432"
  web:
    build: .
    command: > 
        bash -c "
        python manage.py makemigrations
        && python manage.py migrate 
        && python manage.py runserver 0.0.0.0:8000
        "
    image: appserver
# for development purposes, so when we make changes to the source code the change gets saved to the container.
    volumes:
      - .:/appserver
    depends_on:
      - db
    ports:
# expose port 8001 to the host machine
      - "8001:8000"

We're all set now, lets build and run our container using this simple command :

docker-compose up

That’s a simple introduction to how application containers can help your team solve many common packaging, distribution, and operational problems; while helping you ship better software 🤓.

Thanks for the audience and I hope you found this article helpful 🤗. Feel free to reach out to me on Github, Twitter and LinkedIn. Do drop a like, comment, and share 😌.

Further Reading

• Docker-compose documentation
• Docker documentation
• Install docker on ubuntu 20.04

There are situations when the allocation of memory to store data cannot be in a contiguous block of memory or stored sequentially in memory, such situations include When :

• You need constant insertions and deletions from the list (such as in real-time computing where time predictability is absolutely critical).
• You don't know how many items will be in the list.
• You don't need random access to any elements
• You want to be able to insert items in the middle of the list (such as a priority queue) and Et cetera.

We implement data structures such as linked list with nodes. A node is simply a data element which consists of two parts: data & pointer to the next node.

It can consist of address data, geographical data, geometric data, routing information, or transaction details, etc. So nodes help us to know the address of the next data element from the values of the current data element.

Figure 1: A sample implementation of a Single-linked list

These data elements called nodes are linked using pointers, hence called a linked list In Python.

Creation of Nodes for a Single-Linked List

Nodes are the foundations on which various other data structures linked lists and trees can be handled in Python. The nodes are created by implementing a class that will hold the pointers along with the data element.

In the example below, we created a class named "ListNode" to hold the name of Countries. The "next" pointer is initialized to null and three nodes and initialized with values as shown.

The next pointer of node1 points to node3 while the next pointer of node3 points to node2.

class Countries(object):
    """
    the default dataval is set to NIL
    """
    def __init__(self, data=None):
        self.data = data
        self.next = None

"""
 Instantiation of nodes
and setting the data part of the node
to the input value and the next pointer to NIL
"""
node1 = Countries('Nigeria')
node2 = Countries('USA')
node3 = Countries('Kenya')

"""
 Here we're setting the next pointer of node1 to point to node3, and the next pointer of node3 to point to node2
"""
node1.next = node3
node3.next = node2

Traversing the Node Elements

We can traverse the data elements/node of the linked list created above by creating a variable and assigning the first node to it. Then we use a while loop and the next pointer to print out all the node elements.

Note that we have one more additional data element node4 and the next pointers are properly arranged to get the output as countries orderly.

class Countries(object):
    """
    the default dataval is set to NIL
    """
    def __init__(self, data=None):
        self.data = data
        self.next = None

"""
 Instantiation of nodes
and setting the data part of the node
to the input value and the next pointer to NIL
"""
node1 = Countries('Nigeria')
node2 = Countries('USA')
node3 = Countries('Kenya')
node4 = Countries('Canada')

"""
 Here we're setting the next pointer of node1 to point to node3,
 and the next pointer of node3 to point to node2
"""
node1.next = node3
node3.next = node2
node2.next = node4


thisvalue = node1

while thisvalue:
        print(thisvalue.data)
        thisvalue = thisvalue.next

When the above code is executed, the resulting output is given below:

Nigeria
Kenya
USA
Canada

Additional operations like Adding Nodes, Removing Nodes, Searching the list and double-linked lists can be done by implementing appropriate methods using these node containers in the general data structures like linked lists and trees.

There you go with a practical introduction to linked-lists in Python 😀.

Thanks for the audience and I hope you found this article helpful 🤗. Feel free to reach out to me on Github, Twitter and LinkedIn. Do drop a like, comment, and share 😌.