/g/ - Technology

It's was pinoys but same thing tbh

CEO is getting charged for defrauding his investor by using humans while claiming it was AI.

reply to my thread of running llm locally.

>>710

you are not gonna be able to run 67billion parameter model on any laptop (i am not sure about macbooks).

Get rtx 3060, 4060 with highest ram you can manage and then maybe you have chance.

However smaller models 7billion param can in theory be run.

I don't do it, another anon did create a thread about it so find that thread ask him desu.

On that, good idea we should probably start a general dedicated for local models what do you say sirs?

Start by grabbing bits of info from 4chan /g/ and LocalLLaMA

On /g/ there's /aicg/ and /lmg/

>>710

Just run them in google collab. Laptop pe nahi ho payega.

>>711

>>712

yes, yaar, i think some MLfag here should make a dedicated thread to ML which has resources and discussion on ML about new models and research papers.

>>712

I want to actually use it for daily tasks.

https://vicharak.in/axon

I found this on twitter saying that it can run models. if any anon knows kindly elaborate what this thing is.

>>714

>https://vicharak.in/axon

I have been interesting about them for a while, maybe i will get one of their products in future.

I think starting with 6k or somethign not bad.

>>714

>18k rupees

>8gb DDR4

I won't run anything your laptop can't already run.

Seems more like a buffed of raspberry pi, instead of device for running AI models.

>>716

Not sure about price, but there's one product which is like raspberry but not the only one.

>>717

yeah

if you have rtx 3050, decent chunk of ram - i have like 64gb and decent processors you can run many llms with around 7bn parameter or so.

maybe some image generation models like stable diffusion etc.

>>720

well, i have 16GB DDR4, i5 11th gen and rtx 2050 4gb Vram. what can i even do in this? also i think my gpu will work better for anything like this.

>>720

>Simpler or less detailed responses: While a 10B model can handle many tasks very well, it may sometimes lack the richer, more layered answers that come from a model with hundreds of billions of parameters.

>Reduced performance on very complex or highly technical queries: The distilled model might occasionally struggle with the most challenging reasoning tasks compared to its larger counterpart.

what do you think about this or your experience on this?

>>721

try 7bn parameter models - mistral, deepseek, meta all of them probably have those.

iirc deepseek has 1bn parameter model but it's mostly useless

>>722

>what do you think about this or your experience on this?

Give me this week i have decent setup, i can run these models.

I will see some llms and some image gen ones if it works out i will write about it. Probably kickstart a general.

>>723

alright, should i do it on ubuntu or windows? i have 40 Gb left on ubuntu so asking for that only.

>>725

loonix, cuda is more optimized for it and updated

I have separate slightly older setup, if i can run 7bn models on it then ig you will be breeze.

I will test there first. It has limitations same as yours, like 16gb ram etc. etc. Graphics card a bit worse than yours.