ad-data-pf

ad-data-pf

Tables (and later figures) for a research project on job-ad amenities. The functions are written and tested locally against simulated data, then pip installed on and run on the project data..

All numbers in this package come from simulated data.

Use

from ad_data_pf import validation as val, save_tex, save_fig, setup_log

setup_log(log_folder, 'descriptives')      # logs the package version first
frames = {'All AKU\nrespondents': aku, 'All\njob ads': jobads,
          'Linked\njob ads': jobads_linked, 'Job ads linked\nwith AKU response': aku_linked}
save_tex(val.desc_table(frames), out / 'tab_desc.tex')
save_tex(val.omission_table(aku_linked, ci='wilson'), out / 'tab_omission_wilson.tex')
save_fig(val.prevalence_figure(aku_linked, aku), out / 'fig_sensitivity_occ_prevalence.png')

Tables return a bare tabular; figures return a matplotlib Figure. Figures are drawn in style/paper.mplstyle with the colours of style/ggplot2.mplstyle. save_fig(fig, path, slides=True) draws a package figure again in style/slides.mplstyle, for slides. Draw a figure of your own inside with figure_style(): (or figure_style(slides=True)) to match. While a style is in use, matplotlib and fontTools log only warnings and errors. Defaults (variables, labels, panel titles) sit at the top of each module and in ad_data_pf.labels, and can be overridden per call. See the docstrings, and examples/ for every exhibit and for custom tables.

Arguments

Each function fixes the layout of its exhibit: the panels, and the statistic each one computes. The arguments say only which columns go in and what they are called, in three plain forms:

• {column: label}, one row per column: t_vars={'any_T': 'Any non-standard hours', 'work_night': 'Night work'}.
• {column: splits}, one row per group of a column:
  • {code: label} matches the leading characters of a string column ({'1': 'Managers', '2': 'Managers'} on the 6-digit disco) and the value of any other ({True: 'Public', False: 'Private'}). Codes may share a label.
  • [(upper bound, label), ...] bins a numeric column, left-closed, None for no bound (SIZE).
• {label: frame} where the columns, rows or points are samples (desc_table, robustness_table, ladder_figure). The label is the column header, row label or axis label, and \n breaks it into lines. Make each frame with polars before the call: al.filter(c.months_since_hire <= 3), al.with_columns(any_T=c.any_T_sometimes).

Panel titles can be renamed and {} drops a panel. There are no package-specific spec objects and no polars expressions as arguments.

validation makes these exhibits:

Inputs are checked. The following raise an error:

• T or M that is not Boolean, or has nulls;
• a row whose columns exist in no frame;
• a split that no row falls in, or a group with no pairs;
• T and M labels that do not pair up.

Cells resting on fewer than 5 observations are left empty (DST disclosure). In splits, a null or NaN counts as missing: in polars NaN compares above every number, so it would otherwise land in the top bin.

predictive makes the exhibits of text/Notes/20261016_predictive_power_mockup.tex: how well posting topics predict what workers report in AKU, who is hired, and how posting varies within and between workplaces.

Regressions are OLS (linear probability models for Boolean outcomes) by pyfixest, with fixed effects absorbed and standard errors clustered on the workplace (cvrnr x prodnr). A row with a null or NaN in any variable of a regression drops from it, and so do singletons of its fixed effects; the N printed is the N used. The tables report the R2 of the fixed effects alone, the partial R2 of the topics (their share of what the fixed effects leave), and the joint F of the topics. The AUC models are linear probability models cross-fitted by workplace; the full-text one is a LASSO (LassoCV), and categorical variables are target-encoded within each training fold.

Topics are the compacted topics of topic_hierarchy/topics_compaction.py, one Boolean column per topic named as the topic with spaces as underscores (labels.TOPICS). pr.add_flags(df) adds the flags of labels.TOPIC_FLAGS, a flag being any of its topics: the six non-overlapping groups of labels.TOPIC_GROUPS (flexible hours, part-time hours, positive work environment, irregular hours, greedy job, physical work) and the topics matched to AKU variables (labels.OTHER_M_VARS).

The frames, after prep:

On the server:

from ad_data_pf import predictive as pr
from ad_data_pf.labels import AFTER_VARS, HIRE_VARS

jobads = pr.add_flags(jobads)
hires = pr.add_flags(hires).with_columns(mother_u6=c.female & c.child_u6,
                                         father_u6=~c.female & c.child_u6)
al = pr.add_flags(aku_linked).with_columns(mother_u6=c.female & c.child_u6)
pairs = al.filter(c.unique_spell_link_3d)          # the analysis link
save_tex(pr.first_stage_table(pairs, al), out / 'tab_first_stage.tex')
save_tex(pr.gender_table(hires), out / 'tab_gender_hire.tex')
save_fig(pr.shock_figure(jobads, aku, hires), out / 'fig_posting_shocks.png')

examples/predictive/code/run_predictive.py makes every exhibit, including the frame of postings with click data that funnel_figure takes. On the full data auc_figure fits a LASSO 5 times for each of 14 outcomes, on at most max_n=100_000 rows each; lower max_n or folds for a quick run.

CHANGELOG.md lists what to retype when moving to a new version.

Tests on the server data

The tests are installed with the package, and pytest with them. After prep, run them from the server script:

ad_data_pf.run_tests({'aku': aku, 'jobads': jobads,
                      'jobads_linked': jobads_linked, 'aku_linked': aku_linked})

It logs pytest's output as it comes, so it shows on screen and is in the log file of setup_log, and returns 0 if every test passes. The summary at the end names each failed and skipped test, with the reason for a skip. The frames are those after prep with all links kept, named as sim.simulate() names them. They are written to a temporary folder, deleted afterwards, and the tests run in a new process. Further arguments go to pytest: run_tests(frames, '-m', 'data') runs the data tests only, and '-x' stops at the first failure. It skips the test that compiles the tables with latexmk, since LaTeX on the server differs; that test runs locally.

For the predictive tests pass hires and clicks too, and an aku, jobads and aku_linked with the columns those tests need (see the predictive frames above):

ad_data_pf.run_tests({'aku': aku, 'jobads': jobads, 'jobads_linked': jobads_linked,
                      'aku_linked': aku_linked, 'hires': hires, 'clicks': clicks})

Each module's data tests are skipped when a frame they need is not passed.

The data tests (tests/test_validation_data.py, marked data) run on these frames, and on simulated ones when there are none, as in CI. They check what the functions assume of their input but cannot check themselves, and build every exhibit with its defaults:

• T and M flags are Boolean without nulls, and any_T and any_M are the any of the flags in labels.py;
• disco and disco_jobad are 6-digit strings;
• public is Boolean and empl_sum a number of at least 0;
• every pair has a cvrnr and a prodnr, the bootstrap's clusters;
• jobads_linked is part of jobads, and aku_linked of aku and of jobads_linked;
• every default exhibit builds, its counts add up, and its intervals contain its estimates.

tests/test_predictive_data.py checks that:

• every topic of labels.TOPICS is a Boolean column without nulls in jobads, hires and aku_linked (the full-text models would drop a row with a null topic without a word), and wfh_dummy is Boolean;
• the worker reports are Boolean, and varying_hours is null before 2021 and mostly known after;
• the hire's flags are Boolean, its outcomes numbers within a plausible range (a wage in kroner rather than logs fails), and staying 24 months implies staying 12;
• disco and disco_jobad are 6-digit strings, and workplace and year are never null;
• click counts by gender lie between 0 and the total;
• hires is part of jobads, aku_linked of hires, and clicks of jobads;
• every exhibit builds, the first stage without controls is the difference in shares, the four hire types sum to zero, and the R2 of the fixed effects is at most the R2 of the model.

The other tests check the code on fake data. They give the same result anywhere, so on the server they test the environment.

A failed data test shows the offending values: the most common bad codes, or the first rows without a match.

Layout

src/ad_data_pf/
├── functions/          one module per topic: validation.py, predictive.py, ...
│                       imported from the top: from ad_data_pf import validation
├── style/              matplotlib styles: paper, slides, ggplot2 (colours)
├── simulation/         fake data per module (validation.py, predictive.py)
│                       and the files in data/
├── examples/
│   ├── validation/     code/ runs every table on the fake data;
│   │                   logs/ and output/ hold what it writes
│   └── predictive/
└── tests/              test_<module>.py for the code, test_<module>_data.py
                        for what must hold on the server frames

simulation/, examples/ and tests/ are installed with the package. To find them:

python -c "import ad_data_pf, pathlib; print(pathlib.Path(ad_data_pf.__file__).parent)"
python <that folder>/examples/validation/code/run_validation.py <output folder>

Every example also runs in the VS Code Interactive window (or any Jupyter kernel), whole or cell by cell (# %%). It writes to its own logs/ and output/, or to ./ad_data_pf_examples/<name>/ in the working directory if the installed package is read-only.

Adding a module

1. functions/<name>.py: importable at once as ad_data_pf.<name>; arguments in the forms of Arguments
2. simulation/<name>.py: simulate(), save(), load()
3. examples/<name>/code/run_<name>.py, plus logs/.gitkeep. Take the output folder from root = example_root('<name>'), never from sys.argv or __file__, and split sections with # %%. tests/test_examples.py runs every example as a script and as in a Jupyter kernel.
4. tests/test_<name>.py for the code, and tests/test_<name>_data.py, marked data, for what the functions assume of real frames but cannot check themselves (see Tests on the server data). Its frames fixture takes data_folder: sim.load(data_folder), or sim.simulate() when it is None.

Development

The package lives in ad_data_pf/ of the (private) project repository; run everything below from that folder.

uv sync                                          # environment in .venv
uv run pytest                                    # also compiles the LaTeX if latexmk is found
uv run python src/ad_data_pf/simulation/validation.py   # save the fake data to simulation/data
uv run pytest --data src/ad_data_pf/simulation/data     # data tests on saved frames

The lower bounds in pyproject.toml are older versions. To test against exactly those, use a separate environment so uv.lock is not rewritten:

cp uv.lock "$TEMP/uv.lock"
UV_PROJECT_ENVIRONMENT="$TEMP/venv-server" uv sync --resolution lowest-direct
UV_PROJECT_ENVIRONMENT="$TEMP/venv-server" uv run --no-sync pytest
cp "$TEMP/uv.lock" uv.lock

GitHub Actions does both on every push that touches ad_data_pf/ (.github/workflows/ad_data_pf_test.yml at the repository root).

Release

Bump version in pyproject.toml and add an entry to CHANGELOG.md. For each change to a call, give the old and the new call, since the server script is retyped by hand. Commit, then push a tag ad_data_pf-v<version>; GitHub Actions checks that the tag matches the version, runs the tests, builds and publishes to PyPI (trusted publishing, .github/workflows/ad_data_pf_publish.yml):

git tag ad_data_pf-v0.1.1 && git push origin ad_data_pf-v0.1.1

The same tag runs .github/workflows/ad_data_pf_publish_1.yml, which publishes the same code under a second name, ad_data_pf_1. The DST server caches a package when it is installed and won't update that name for 24 hours. On the server, install an exact version under whichever name hasn't been installed in the last 24 hours:

pip install ad-data-pf==X.Y.Z
pip install ad_data_pf_1==X.Y.Z

Both install the import ad_data_pf, and the most recent install is the one used.