Enhance pipeline stability and automatically retry unstable tests

[vsychov]

Hello there,

This Pull Request aims to introduce changes designed to enhance the overall stability of the pipeline and implement an automatic retry mechanism for tests that are currently unstable.

There is example, pipeline with retry: vsychov#2
There is pipeline without retry: #1562

Both of them contains same headscale file changes.

Another way to do that, is use --rerun-fails and --rerun-fails-max-failures arguments of gotestsum.

• read the CONTRIBUTING guidelines

[vsychov]

@kradalby, can you take a look?

[kradalby]

Apologise, missed this one, I had it in my todo list to do something like this, but I think 50 is a bit excessive, how about we start with 5.

The goal needs to be to get the tests to a state where that (or less) is the upper threshold.

[vsychov]

@kradalby , done, I changed it to 5.

[kradalby]

I think this removes a bit too much?

Suggested change

	with:
	attempt_limit: 5
	command: |
	with:
	attempt_limit: 5
	command: |
	nix develop --command -- docker run \

[vsychov]

Yes, sorry for that, I didn't check the changes after rebasing to main, now it's fixed.