StepRight - Unit Testing

Section 5’s job proves the pipeline runs; nothing yet proves the transformation logic inside it is correct, independent of whatever data happens to be sitting in dev.step_right on a given day. This section fixes that: a handful of business-critical calculations – revenue, discount allocation, deduplication, the quarantine threshold itself – get extracted into pure, DataFrame-in/DataFrame-out functions and covered by a pytest suite that runs in seconds, on a laptop, with no cluster and no live workspace required.

flowchart LR
    subgraph Before["Inside @dp.table functions"]
        A[Revenue math] 
        B[Dedup logic]
        C[Threshold decision]
    end
    subgraph After["Extracted, pure functions"]
        A2[gold_logic.py]
        B2[silver_logic.py]
        C2[dq_logic.py]
    end
    A -->|Lecture 2| A2
    B -->|Lecture 2| B2
    C -->|Lecture 4| C2
    A2 & B2 & C2 --> Tests[pytest suite<br/>local SparkSession, no cluster]
    Tests -->|green| CI[Section 8: runs on every PR]

Lectures

#	Lecture	Est. Read
1	Design Test Strategy for the Project	11 min read
2	Refactoring Your Code and Getting Ready for Unit Testing	7 min read
3	Designing and Developing Unit Test Cases	17 min read
4	Code and Run all Unit Test Case	12 min read

---

| ← Previous: Creating the Orchestration Job | Next: Design Test Strategy for the Project → | |:—|—:|

---

Table of contents

• Design Test Strategy for the Project
• Refactoring Your Code and Getting Ready for Unit Testing
• Designing and Developing Unit Test Cases
• Code and Run all Unit Test Case