Description
generate_projects_json has two fallback paths when LLM project selection fails, and they behave differently:
- JSON decode error (L435-L440) returns
projects_data[:7] — the list already filtered to exclude repos with author_commit_count == 0, with all fields intact.
- Any other exception (L442-L460) rebuilds from the unfiltered
projects[:7]:
https://github.com/interviewstreet/hiring-agent/blob/70fd3ea/github.py#L442-L460
except Exception as e:
print("🔄 Falling back to first 7 projects")
projects_data = []
for project in projects[:7]: # unfiltered
project_data = { ... } # no author_commit_count / total_commit_count
The second path is hit frequently in practice: provider errors (429/5xx after retries, missing API key, network errors) and responses that parse as JSON but aren't a list of dicts, e.g. {"projects": [...]}. Iterating a dict yields string keys, and project.get raises AttributeError.
Result: the evaluator can receive repositories the candidate never committed to (e.g. starred-and-forked popular repos with forks_count >= 5), and without the commit counts it cannot tell. This inflates the open source score.
Expected
Both fallbacks use the same filtered projects_data list with all fields, e.g. build projects_data before the try and return projects_data[:7] in both handlers.
Description
generate_projects_jsonhas two fallback paths when LLM project selection fails, and they behave differently:projects_data[:7]— the list already filtered to exclude repos withauthor_commit_count == 0, with all fields intact.projects[:7]:https://github.com/interviewstreet/hiring-agent/blob/70fd3ea/github.py#L442-L460
The second path is hit frequently in practice: provider errors (429/5xx after retries, missing API key, network errors) and responses that parse as JSON but aren't a list of dicts, e.g.
{"projects": [...]}. Iterating a dict yields string keys, andproject.getraisesAttributeError.Result: the evaluator can receive repositories the candidate never committed to (e.g. starred-and-forked popular repos with
forks_count >= 5), and without the commit counts it cannot tell. This inflates the open source score.Expected
Both fallbacks use the same filtered
projects_datalist with all fields, e.g. buildprojects_databefore thetryandreturn projects_data[:7]in both handlers.