Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

268 bug the title 2 total values and hence the overall total values returned by the all programs endpoint needs to be corrected #269

Merged
merged 7 commits into from
Nov 5, 2024
Merged

268 bug the title 2 total values and hence the overall total values returned by the all programs endpoint needs to be corrected #269

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
9f8366d
When calculating total benefit payment, in CRP, include only Total CR…
sandeep-ps Nov 1, 2024
3489292
Update CHANGELOG.
sandeep-ps Nov 1, 2024
200b6a4
Update README with detail on total payment amount calculation.
sandeep-ps Nov 1, 2024
ac75582
Add total for year range to state JSON object.
sandeep-ps Nov 4, 2024
7cd12a7
Use start and end years to calculate key and minor changes to comments.
sandeep-ps Nov 4, 2024
4812d9f
Fix SQL query for total payment calculation by including Title-1 sub …
sandeep-ps Nov 4, 2024
9bd665a
Simplify the SQL queries.
sandeep-ps Nov 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,14 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [0.18.0] - 2024-10-31
## [0.19.0] - 2024-11-03

### Changed
- Landing page values to use data from the database. [#258](https://github.com/policy-design-lab/pdl-api/issues/258)

### Fixed
- Title-II total payment calculation and overall benefits payment amounts calculation in the GET /pdl/allprograms endpoint. [#268](https://github.com/policy-design-lab/pdl-api/issues/268)

## [0.18.0] - 2024-10-07

### Changed
Expand Down
5 changes: 4 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,4 +44,7 @@ Now you can view the API at http://localhost/pdl/

For swagger doc, go to http://localhost/ui


## Short note on total payment calculation
When calculating the total dollar amounts payment calculation, the following details are considered in the GET /pdl/allprograms endpoint:
1. In CRP, the "Total CRP" sub program contains the total value of all CRP payments. Just for the purpose of calculating the total values, the other sub programs and sub-sub programs in that can be ignored.
2. In Crop Insurance, the column `net_farmer_benefits_amount` contain the amount value that should be used for the total payment calculation. The `payment` column value is NULL for that program.
56 changes: 22 additions & 34 deletions app/controllers/pdl.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -762,39 +762,20 @@ def generate_allprograms_response(start_year, end_year):

for year in range(start_year, end_year + 1):
year_columns.append(f"""
-- Title I: Commodities
SUM(CASE WHEN t.name = 'Title I: Commodities' AND p.year = {year} THEN p.payment ELSE 0 END)
AS "Title I {year}",
-- Title II: Conservation
SUM(CASE WHEN t.name = 'Title II: Conservation' AND p.year = {year} THEN p.payment ELSE 0 END)
AS "Title II {year}",
-- Title IV: SNAP
SUM(CASE WHEN t.name = 'Title IV: Nutrition' AND p.year = {year} THEN p.payment ELSE 0 END)
AS "SNAP {year}",
-- Title IX: Crop Insurance
SUM(CASE WHEN t.name = 'Title IX: Crop Insurance' AND p.year = {year}
THEN p.net_farmer_benefit_amount ELSE 0 END) AS "Crop Insurance {year}",
-- All Programs Total for {year}
SUM(CASE WHEN p.year = {year} THEN COALESCE(p.payment, 0) + COALESCE(p.net_farmer_benefit_amount, 0)
ELSE 0 END) AS "{year} All Programs Total"
SUM(CASE WHEN t.name = 'Title I: Commodities' AND p.year = {year} THEN p.payment ELSE 0 END) AS "Title I {year}",
SUM(CASE WHEN t.name = 'Title II: Conservation' AND p.year = {year} AND (sub_program_id IN (SELECT id FROM pdl.sub_programs WHERE pdl.sub_programs.name = 'Total CRP') OR sub_program_id IS NULL) THEN p.payment ELSE 0 END) AS "Title II {year}",
SUM(CASE WHEN t.name = 'Title IV: Nutrition' AND p.year = {year} THEN p.payment ELSE 0 END) AS "SNAP {year}",
SUM(CASE WHEN t.name = 'Title IX: Crop Insurance' AND p.year = {year} THEN p.net_farmer_benefit_amount ELSE 0 END) AS "Crop Insurance {year}",
SUM(CASE WHEN p.year = {year} AND (sub_program_id IN (SELECT id FROM pdl.sub_programs WHERE pdl.sub_programs.name IN ('Total CRP', 'Agriculture Risk Coverage County Option (ARC-CO)', 'Agriculture Risk Coverage Individual Coverage (ARC-IC)')) OR sub_program_id IS NULL) THEN COALESCE(p.payment, 0) + COALESCE(p.net_farmer_benefit_amount, 0) ELSE 0 END) AS "{year} All Programs Total"
""")

# Add the totals for each program and the overall total
totals = f"""
-- Corrected total for Title I
SUM(CASE WHEN t.name = 'Title I: Commodities' AND p.year BETWEEN {start_year} AND {end_year}
THEN p.payment ELSE 0 END) AS "Title I Total",
-- Corrected total for Title II
SUM(CASE WHEN t.name = 'Title II: Conservation' AND p.year BETWEEN {start_year} AND {end_year}
THEN p.payment ELSE 0 END) AS "Title II Total",
-- Corrected total for SNAP
SUM(CASE WHEN t.name = 'Title IV: Nutrition' AND p.year BETWEEN {start_year} AND {end_year}
THEN p.payment ELSE 0 END) AS "SNAP Total",
-- Corrected total for Crop Insurance
SUM(CASE WHEN t.name = 'Title IX: Crop Insurance' AND p.year BETWEEN {start_year} AND {end_year}
THEN p.net_farmer_benefit_amount ELSE 0 END) AS "Crop Insurance Total",
-- 18-22 All Programs Total
SUM(COALESCE(p.payment, 0) + COALESCE(p.net_farmer_benefit_amount, 0)) AS "18-22 All Programs Total"
SUM(CASE WHEN t.name = 'Title I: Commodities' AND p.year BETWEEN {start_year} AND {end_year} THEN p.payment ELSE 0 END) AS "Title I Total",
SUM(CASE WHEN t.name = 'Title II: Conservation' AND p.year BETWEEN {start_year} AND {end_year} AND (sub_program_id IN (SELECT id FROM pdl.sub_programs WHERE pdl.sub_programs.name = 'Total CRP') OR sub_program_id IS NULL) THEN p.payment ELSE 0 END) AS "Title II Total",
SUM(CASE WHEN t.name = 'Title IV: Nutrition' AND p.year BETWEEN {start_year} AND {end_year} THEN p.payment ELSE 0 END) AS "SNAP Total",
SUM(CASE WHEN t.name = 'Title IX: Crop Insurance' AND p.year BETWEEN {start_year} AND {end_year} THEN p.net_farmer_benefit_amount ELSE 0 END) AS "Crop Insurance Total",
SUM(CASE WHEN (sub_program_id IN (SELECT id FROM pdl.sub_programs WHERE pdl.sub_programs.name IN ('Total CRP', 'Agriculture Risk Coverage County Option (ARC-CO)', 'Agriculture Risk Coverage Individual Coverage (ARC-IC)')) OR sub_program_id IS NULL) THEN COALESCE(p.payment, 0) + COALESCE(p.net_farmer_benefit_amount, 0) ELSE 0 END) AS "{str(start_year)[-2:]}-{str(end_year)[-2:]} All Programs Total"
"""

# Combine the query of non-dynamic part
Expand All @@ -819,6 +800,7 @@ def generate_allprograms_response(start_year, end_year):

state_data = []
total_row = {"State": "Total"}
start_to_end_years_total_key = str(start_year)[-2:] + "-" + str(end_year)[-2:] + " All Programs Total"

# Initialize total_row with zero values for all keys in the desired order
for year in range(start_year, end_year + 1):
Expand All @@ -839,7 +821,7 @@ def generate_allprograms_response(start_year, end_year):

for year in range(start_year, end_year + 1):
total_row[f"{year} All Programs Total"] = 0
total_row["18-22 All Programs Total"] = 0
total_row[start_to_end_years_total_key] = 0

for row in result:
result_dict = dict(zip(columns, row)) # Using zip() to match columns with their values
Expand Down Expand Up @@ -896,12 +878,15 @@ def generate_allprograms_response(start_year, end_year):
total_row[f"{year} All Programs Total"] += value

# Add the final total for all programs between the years
formatted_result["18-22 All Programs Total"] = result_dict.get("18-22 All Programs Total", 0)
total_row["18-22 All Programs Total"] += formatted_result["18-22 All Programs Total"]
formatted_result[start_to_end_years_total_key] = result_dict.get(start_to_end_years_total_key, 0)

# Append the formatted result to the state data
state_data.append(formatted_result)

# Add the final total for all programs between the years
for year in range(start_year, end_year + 1):
total_row[start_to_end_years_total_key] += total_row[f"{year} All Programs Total"]

# Append the total row at the end of state data
state_data.append(total_row)

Expand Down Expand Up @@ -931,16 +916,19 @@ def generate_summary_response(start_year, end_year):
pdl.payments p
JOIN
pdl.titles t ON p.title_id = t.id
LEFT JOIN
pdl.sub_programs sp ON p.sub_program_id = sp.id
WHERE
p.year BETWEEN {start_year} AND {end_year}
AND (sp.name IN ('Total CRP', 'Agriculture Risk Coverage County Option (ARC-CO)', 'Agriculture Risk Coverage Individual Coverage (ARC-IC)') OR p.sub_program_id IS NULL)
GROUP BY
t.name, p.state_code, p.year
ORDER BY
CASE
WHEN t.name = 'Title I: Commodities' THEN 1
WHEN t.name = 'Title II: Conservation' THEN 2
WHEN t.name = 'Title IX: Crop Insurance' THEN 3
WHEN t.name = 'Title IV: Nutrition' THEN 4 -- SNAP comes last
WHEN t.name = 'Title IV: Nutrition' THEN 4
END,
p.year,
p.state_code;
Expand Down Expand Up @@ -984,7 +972,7 @@ def generate_summary_response(start_year, end_year):

except Exception as e:
# Log any errors that occur
logger.error(f"Error occurred: {str(e)}")
logging.error(f"Error occurred: {str(e)}")
return {"error": str(e)}

finally:
Expand Down
Loading