Fix goroutine leak in EphemeralGC on node cancel

[qoke]

There is a goroutine leak in EphemeralGarbageCollector where each call to Schedule() creates a goroutine, but those goroutines are not properly terminated when a node is canceled or when the garbage collector is shut down i.e. leaving dangling goroutines.

Wrote a test to confirm:

=== RUN TestEphemeralGarbageCollectorGoRoutineLeak
ephemeral_garbage_collector_test.go:16: Initial number of goroutines: 2
ephemeral_garbage_collector_test.go:68: Final number of goroutines: 102
ephemeral_garbage_collector_test.go:71:
Error Trace: github.com/juanfont/headscale/hscontrol/db/ephemeral_garbage_collector_test.go:71
Error: "102" is not less than or equal to "7"
Test: TestEphemeralGarbageCollectorGoRoutineLeak
Messages: There are significantly more goroutines after GC usage, which suggests a leak
--- FAIL: TestEphemeralGarbageCollectorGoRoutineLeak (0.30s)
FAIL
FAIL github.com/juanfont/headscale/hscontrol/db 0.317s
FAIL

The fix ensures a proper cleanup of dangling goroutines and stopping all timers. Also added a safety check in Schedule() to prevent scheduling after Close() - There don't appear to be any current codepaths that call in this order currently, but I have added a hygiene check for this to ensure if someone does this in future, that it is handled correctly.

After this fix, also added a bunch of additional tests to ensure cancel and reschedule are not caught up in the same problem (which they are not).

=== RUN TestEphemeralGarbageCollectorGoRoutineLeak
ephemeral_garbage_collector_test.go:21: Initial number of goroutines: 2
ephemeral_garbage_collector_test.go:71: Final number of goroutines: 2
--- PASS: TestEphemeralGarbageCollectorGoRoutineLeak (0.50s)
=== RUN TestEphemeralGarbageCollectorReschedule
--- PASS: TestEphemeralGarbageCollectorReschedule (0.10s)
=== RUN TestEphemeralGarbageCollectorCancelAndReschedule
--- PASS: TestEphemeralGarbageCollectorCancelAndReschedule (0.20s)
=== RUN TestEphemeralGarbageCollectorCloseBeforeTimerFires
--- PASS: TestEphemeralGarbageCollectorCloseBeforeTimerFires (0.10s)
=== RUN TestEphemeralGarbageCollectorScheduleAfterClose
ephemeral_garbage_collector_test.go:205: Initial number of goroutines: 2
ephemeral_garbage_collector_test.go:248: Final number of goroutines: 2
--- PASS: TestEphemeralGarbageCollectorScheduleAfterClose (0.20s)
=== RUN TestEphemeralGarbageCollectorConcurrentScheduleAndClose
ephemeral_garbage_collector_test.go:260: Initial number of goroutines: 2
ephemeral_garbage_collector_test.go:331: Final number of goroutines: 2
--- PASS: TestEphemeralGarbageCollectorConcurrentScheduleAndClose (0.35s)
=== RUN TestEphemeralGarbageCollectorOrder
--- PASS: TestEphemeralGarbageCollectorOrder (7.00s)
=== RUN TestEphemeralGarbageCollectorLoads
--- PASS: TestEphemeralGarbageCollectorLoads (10.01s)
PASS
ok github.com/juanfont/headscale/hscontrol/db 18.487s

Given the amount of time these tests take to run, you might not want to run these by default.

[ghost]

Pull Request Revisions

Revision	Description
r6	Fixed typo in garbage collector comment Corrected a minor typo in the comment explaining the logic of the ephemeral garbage collector's scheduling mechanism
r5	Improved node deletion channel handling Added more robust channel communication for node deletion in garbage collector, preventing potential race conditions during shutdown
r4	Added constants for time durations Introduced named constants for millisecond time durations to improve code readability in ephemeral garbage collector test file
r3	Improved ephemeral garbage collector test Refactored ephemeral garbage collector tests with improved synchronization, channel-based signaling, and more robust goroutine leak detection
r2	Improved goroutine leak prevention logic Enhanced ephemeral garbage collector to prevent potential goroutine leaks by adding cancellation channel checks and improving error handling
r1	Improved ephemeral garbage collector implementation Enhanced ephemeral garbage collector with more robust concurrency handling, safer Close() method, and improved Schedule() logic with better timer management

✅ AI review completed for r6

Help

React with emojis to give feedback on AI-generated reviews:

• 👍 means the feedback was helpful and actionable
• 👎 means the feedback was incorrect or unhelpful

💬 Replying to feedback with a comment helps us improve the system. Your input also contributes to shaping future interactions with the AI reviewer.

We'd love to hear from you—reach out anytime at [email protected].

[kradalby]

Looks great! thank you!