8
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
this post was submitted on 09 Aug 2023
8 points (78.6% liked)
Selfhosted
40219 readers
774 users here now
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.
Rules:
-
Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
-
No spam posting.
-
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.
-
Don't duplicate the full text of your blog or github here. Just post the link for folks to click.
-
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
-
No trolling.
Resources:
- selfh.st Newsletter and index of selfhosted software and apps
- awesome-selfhosted software
- awesome-sysadmin resources
- Self-Hosted Podcast from Jupiter Broadcasting
Any issues on the community? Report it using the report flag.
Questions? DM the mods!
founded 1 year ago
MODERATORS
Stop. Go back. This is the wrong way.
If you're running python you basically need a full os.
There are projects that run as an rtos, and in fact I worked on an ml soc that ran Linux, but there are 2 levels here:
The ml processing itself, ie the math. This is simple in software and very complex otherwise. The software just says "copy this block and start running a matrix multiply". The hard logic is in moving data around efficiently.
The stack. This is high level, python or so, and has graph processing overhead too. This needs a lot of "overhead" by its nature.
In either case, don't worry about any of this, the overhead won't be very noticeable, you'll be cpu gated hard, the main thing is finding an optimized pytorch library.
If you have an amd cpu or somehow have an nvidia gpu in your laptop you might be able to use their pytorch library which would improve performance by roughly 1.5-2 orders of magnitude.
Unfortunately there isn't a pytorch implementation for Intel igpus, but there is an opencl backend for pytorch, and apparently this madlad got it working through opencl on an Intel igpu: https://dev-discuss.pytorch.org/t/implementing-opencl-backend-for-pytorch/283/9
But don't worry about overhead, it's less than fractions of percents in these kinds of tasks and there are ways to bypass them completely.
Seems like avoiding context switching and all the overhead associated would make a big difference when pretty much everything in cache is critical data.
I'm more curious about running something like Bark TTS where the delay is not relevant, but it would be cool to have the instructional clarity of the Halo Master Chief voice read me technical documentation, or test the effects of training my own voice, tuned to how I hear it, reading me stuff I find challenging. If the software is only able to process around 10 seconds at a time, just script it and let it run. The old machine will just collect dust otherwise.
Anyways, what's the best scheduler with affinity/isolation/pinning?
It's not. Like the commenter above said: It's a fraction of the task at hand. Especially when you design the rest of the system to run only if necessary. Context Switches are what? like 50 CPU Cycles? Store Registers, Store TCB, Load other TCB and load other register states jump back to PC. Maybe some other OS Shenanigans, but that's basically it.
Now Imagine complex calculations on a 25-Dimensional Matrix.