MinIO: Major Refactor Release

## Highlights This is a major refactor release with WebSockets for internal communication between nodes. We advise our users to follow our upgrade instructions on best [practices](https://min.io/docs/minio/linux/operations/install-deploy-manage/upgrade-minio-deployment.html#upgrade-a-minio-deployment) - Click [here](https://min.io/download) to download the latest version of MinIO. - Click [here](https://min.io/pricing) for production support. - This release brings a larger change to internode communication, which is now re-implemented to use WebSockets for most of the internal calls. Allowing us much larger scalability to **100s** to **1000s** of nodes. To learn more about the WebSockets grid implementation [please read](https://github.com/minio/minio/blob/master/internal/grid/README.md?plain=1). - Site replication now supports optionally heal ILM configuration to all sites, allowing ILM settings across buckets to be centrally managed via any site. - Batch replication from minio -> minio ultimately uses compressed archive by default to more effectively utilize the bandwidth during batch replication. - Support for Batch expiration of objects on your MinIO instance. - MinIO now also supports external caching for metadata calls such as HeadObject() in a distributed memory for faster lookups when you have a slower medium such as HDD and a READ-heavy workload pattern. ## What's Changed * Websocket grid functionality by @klauspost in https://github.com/minio/minio/pull/18461 * return errors if dataDir is missing during HeadObject() by @harshavardhana in https://github.com/minio/minio/pull/18477 * fix: close http body for es action by @jiuker in https://github.com/minio/minio/pull/18491 * Replicate Expiry ILM configs while site replication by @shtripat in https://github.com/minio/minio/pull/18130 * use Access(F_OK) do not need to check for permissions by @harshavardhana in https://github.com/minio/minio/pull/18492 * fix: go mod was pointed to personal repos with replace remove it by @harshavardhana in https://github.com/minio/minio/pull/18498 * Adding a fix to check for bucket existence before upload by @zveinn in https://github.com/minio/minio/pull/18496 * use the latest UBI image by @harshavardhana in https://github.com/minio/minio/pull/18497 * Implement batch snowball by @vadmeste in https://github.com/minio/minio/pull/18485 * Healing tweaks by @klauspost in https://github.com/minio/minio/pull/18225 * ilm: ObjectSizeLessThan and ObjectSizeGreaterThan by @krisis in https://github.com/minio/minio/pull/18500 * feat: bring new HDD related performance enhancements by @harshavardhana in https://github.com/minio/minio/pull/18239 * snowball-repl: Add support of immediate tiering by @vadmeste in https://github.com/minio/minio/pull/18508 * DiskInfo() must return errDiskNotFound not internal errors by @harshavardhana in https://github.com/minio/minio/pull/18514 * Relax batch size limit for kafka events by @Praveenrajmani in https://github.com/minio/minio/pull/18513 * No shallow copy needed by @shtripat in https://github.com/minio/minio/pull/18499 * Pass SUBNET URL to console by @anjalshireesh in https://github.com/minio/minio/pull/18503 * Add detailed parameter tracing + custom prefix by @klauspost in https://github.com/minio/minio/pull/18518 * remove frivolous logging in transition object by @harshavardhana in https://github.com/minio/minio/pull/18526 * feat: drive subsys to max timeout reload by @jiuker in https://github.com/minio/minio/pull/18501 * support passing signatureAge conditional by @harshavardhana in https://github.com/minio/minio/pull/18529 * avoid Walk() API listing objects without quorum by @harshavardhana in https://github.com/minio/minio/pull/18535 * xl: Cleaning up will delete instead of rename to trash with full disk err by @vadmeste in https://github.com/minio/minio/pull/18534 * fix(cmd/notification.go): no error when retry successful by @bestgopher in https://github.com/minio/minio/pull/18530 * fix: delete null version at suspend version by @jiuker in https://github.com/minio/minio/pull/18547 * Skip versions expired by DeleteAllVersionsAction by @krisis in https://github.com/minio/minio/pull/18537 * Fix shadowed error by @klauspost in https://github.com/minio/minio/pull/18548 * Fixed transition rules getting overwritten while healing by @shtripat in https://github.com/minio/minio/pull/18542 * Improve env var config error reporting by @klauspost in https://github.com/minio/minio/pull/18549 * Always remove subroute when queuing message on the connection. by @klauspost in https://github.com/minio/minio/pull/18550 * (chore): update all our deps by @harshavardhana in https://github.com/minio/minio/pull/18525 * xl: Rename Force to Immediate for clarity by @vadmeste in https://github.com/minio/minio/pull/18540 * cleanup handling of STS isAllowed and simplifies the PolicyDBGet() by @harshavardhana in https://github.com/minio/minio/pull/18554 * feat: support es compression by @jiuker in https://github.com/minio/minio/pull/18562 * site replication must heal policy mappings with correct userType by @harshavardhana in https://github.com/minio/minio/pull/18563 * Add TLS cert checksum by @klauspost in https://github.com/minio/minio/pull/18557 * allow hotfixes to generate deb, rpm packages by @harshavardhana in https://github.com/minio/minio/pull/18568 * Fix Mux Connect Error by @klauspost in https://github.com/minio/minio/pull/18567 * skip ILM expired objects from healing by @harshavardhana in https://github.com/minio/minio/pull/18569 * Local and Remote hosts swapped in grid traces by @klauspost in https://github.com/minio/minio/pull/18574 * Fix drives fixed after startup by @klauspost in https://github.com/minio/minio/pull/18576 * Minor fixes to bucket replication by @krisis in https://github.com/minio/minio/pull/18578 * Disable DMA optimization on windows by @klauspost in https://github.com/minio/minio/pull/18575 * Implement batch-expiration for objects by @krisis in https://github.com/minio/minio/pull/17946 ## New Contributors * @bestgopher made their first contribution in https://github.com/minio/minio/pull/18530 **Full Changelog**: https://github.com/minio/minio/compare/RELEASE.2023-11-20T22-40-07Z...RELEASE.2023-12-02T10-51-33Z