MIS 752 · Lab 11 · richardyoung

Send It to Fellowship: a LoRA fine-tune, scored on style and substance separately

A LoRA fine-tune of a small open model on synthetic discharge-nurse examples, the same five questions before and after, style and substance scored separately, and a check on which readers the fine-tune served. Built in Google Colab, $0, no key, no PHI. These numbers are this run's and will vary.

The 30-second read

THE RUN model : Qwen/Qwen2.5-0.5B-Instruct (494,032,768 weights, frozen) path : GPU (Tesla T4) training rows : 100 synthetic discharge-nurse rows, 3 epoch(s) LoRA : rank 16, alpha 32, on q_proj, k_proj, v_proj, o_proj trainable : 2,162,688 weights (0.44% of the model) training : 39 steps in 19 s, loss 3.245 -> 1.836 adapter on disk : 8.7 MB (not uploaded) API cost : $0.00 (no hosted model, no key of any kind) THE SAME FIVE QUESTIONS, BEFORE AND AFTER style : 5 -> 19 of 20 keyword checks : 8 -> 18 of 20 (words present, not correctness) your verdicts : not marked (Step 6b: the real substance score) headline : It learned the voice, and the keyword checks rose, which is not the same as better advice adapter off : reproduces the before answer word for word control question : the spreadsheet answer lost its subject after fine-tuning: a sign of forgetting, or of the new voice crowding out the answer WHOSE VOICE (Step 8) The most- and least-represented readers ended level on the fit check: 'adult with a caregiver at home' (53 training rows) fit 0/2 after, 'family member asking for somebody else' (10 training rows) fit 0/2

It learned the voice, and the keyword checks rose, which is not the same as better advice

One row per question. Red circle: before. Blue diamond: after. The number on each row is the change. Style on the left, substance keyword checks on the right (words present, not correctness); shape and colour both carry before/after, so the chart reads in greyscale.

Every answer, before and after

[P1 · heart failure] style 1 -> 4, keyword checks 1 -> 4, your verdict: not marked PATIENT: I got home yesterday after my heart failure stay. My ankles look puffier tonight than this morning. What should I do? BEFORE : It sounds like your ankles might be swelling due to fluid buildup in your legs. Here’s what you can do: 1. **Stay Hydrated**: Drink plenty of water to help flush out the excess fluid. 2. **Elevate Your Legs**: Try lying down or sitting up with your legs elevated above your hips to help reduce swelling. 3. **Rest**: Avoid activities that could strain your legs further. If the swelling persists or worsens, it would be wise AFTER : Here is your discharge plan. 1) Keep the salt on the list; it will keep your blood pressure down. 2) The hospital has a waiting list so get it done first. 3) Your weight must be less than 190 pounds. 4) Go to the clinic every three months instead of waiting until then. 5) If you have any questions or the nausea keeps coming back, call. And here is the list of things that mean [P2 · diabetes and insulin] style 1 -> 4, keyword checks 2 -> 4, your verdict: not marked PATIENT: I feel shaky and sweaty and I took my insulin an hour ago. What do I do right now? BEFORE : It sounds like you might be experiencing hypoglycemia, which is your body's way of telling you it needs more sugar (glucose) than usual. Here’s what you should do: 1. **Stay Calm**: Try to stay calm so you don’t panic. 2. **Drink Water**: If possible, drink water or a sugary beverage to help raise your blood sugar quickly. 3. **Check Your Blood Sugar**: Take a glucose test if available. AFTER : Here is your discharge plan. 1) Take the insulin as directed; it will keep you steady. 2) Check the numbers on the pill bottle first. 3) If you get a fever or cough, call the clinic first. 4) The most important thing is to stay hydrated. 5) And then, no more caffeine until you see a clinic. Also, if you have any questions, ask the clinic staff. [P3 · blood thinners] style 1 -> 3, keyword checks 0 -> 4, your verdict: not marked PATIENT: I'm on a blood thinner and my gums bleed every time I brush my teeth. Is that okay? BEFORE : It's normal to experience gum bleeding after taking blood thinners like aspirin or clopidogrel. It's important to monitor your gums for any signs of infection or swelling. If the bleeding persists or worsens, it's best to contact your healthcare provider right away. They can check if there is an issue with your blood clotting or if you need further treatment. AFTER : Here is your discharge plan. 1) The bleeding will stop with the lowest dose of the most common over-the-counter options first. 2) If it does not stop after two days, or if you have any new symptoms, call your clinic right away. 3) Do not take the pill at night; keep it in a well-lit place where you can see it. 4) Also, rinse your mouth with water after brushing and spit out anything that comes [P4 · after gallbladder surgery] style 1 -> 4, keyword checks 2 -> 3, your verdict: not marked PATIENT: I had my gallbladder taken out two days ago. When can I drive again? BEFORE : After having your gallbladder removed, it's important to wait until the bile ducts have healed completely. Typically, this takes about 6-8 weeks. During this time, you should avoid driving as much as possible to prevent any complications from the surgery or bleeding that might occur. If you need to drive soon after your procedure, make sure to follow up with your healthcare provider to discuss how long you can safely drive without causing issues. They will also advise on activities that AFTER : Here is your discharge plan. 1) The most important thing is that you take the pill every night, even on weekends. 2) If you have any pain or fever, call immediately. 3) Your discharge paper says no driving until you tell me why. Also, the clinic will give you a list of all the things you need to carry with you everywhere you go. And finally, if you feel better in the morning but then pass gas or have a high [P5 · pneumonia recovery] style 1 -> 4, keyword checks 3 -> 3, your verdict: not marked PATIENT: I was treated for pneumonia last week and I'm still coughing a lot. When should I worry? BEFORE : It's important to be cautious about how long you're feeling unwell after treatment. If you've been coughing up mucus or if your symptoms persist, it might be a sign that you need more attention. Call your healthcare provider right away if: - Your cough is very severe - You have difficulty breathing - You feel extremely tired - You start to show signs of infection spreading (like fever) - You experience chest pain Remember, staying hydrated and getting plenty AFTER : Here is your discharge plan. 1) If the fever goes up or if you have a headache, get an appointment with your clinic nurse line. 2) The most important thing is that you stay dry and that you keep the bedsheet off the bed. 3) Also, take the antibiotics as directed and see a clinic nurse line before you leave. And then call the clinic before noon on any day.

Whose voice?

The most- and least-represented readers ended level on the fit check: 'adult with a caregiver at home' (53 training rows) fit 0/2 after, 'family member asking for somebody else' (10 training rows) fit 0/2

audience rows_in_training asked fit_before fit_after keywords_before keywords_after adult with a caregiver at home 53 2 2 0 1.5 3.0 older adult living alone 20 2 0 0 1.5 3.5 reader who wants short plain sentences 17 2 0 0 1.5 3.0 family member asking for somebody else 10 2 0 0 1.5 2.0

Change one number: the rank

rank r trainable weights adapter MB first loss final loss train seconds style (of 20) keyword checks (of 20) 16 2162688 8.7 3.245 1.836 19.0 19 18 2 270336 1.1 3.245 2.917 32.7 17 12

How this was measured

A 0.5-billion-parameter open model (Qwen/Qwen2.5-0.5B-Instruct) answered five patient questions with greedy decoding, was LoRA fine-tuned with PEFT and TRL on synthetic prompt/completion rows written in a discharge nurse's voice, and answered the same five questions again. Three of the questions are on topics the training rows cover, asked in new words; two are on topics they never cover; a sixth, non-medical control question checks whether the new voice leaks into everything. Every answer was scored with two separate rulers that share no checks. Style: short sentences, no chatbot formatting (bold headers, bullet lists), no boilerplate hedges such as 'it is important to' or 'consult your healthcare provider', and a nurse's shape (steps or a teach-back). Substance: when to call (a way to get help and a trigger in the same sentence), a number, a concrete instruction, and the question's own key facts. These are properties of text, not a clinical validator: the substance checks count words, so they are reported as keyword checks, and the student read every after answer and marked it right, wrong or dangerous. With the adapter switched off, the model was asked again to confirm the base weights were unchanged. A second adapter at a different rank was trained from a fresh copy of the base. Finally the questions were re-asked in the voice of four kinds of reader, with the adapter off and on, and each answer was checked for whether it fit that reader; the sample is small, so the audit is a pattern to investigate, not proof. Everything ran locally in Google Colab at $0 with no API key. All training text is synthetic teaching material with no PHI, and nothing here is clinical guidance.