From scratch, then get hold of Charles Petzold's book Code .
Unfortunately Code wasn't written when I was doing wire wraps on breadboards try to build a half adder using bc109s (or what ever transitors we were using) -if it had been the cybernetics weekend I spent as a teenager might have made a whole lot more sense.
Petzold starts right at the beginning with light bulbs and switches, spends a long time with relays and ends up with those classic microprocessors the 8080 and the 6800. If you stick with him, not a hard task, then you will understand how a modern computer works. I haven't quite finished it, but the book is restricted to classic von Neuman machines, I haven't seen any discussion of massively parallel architectures, RISC processors or data-flow engines, never mind quantum computers, but you can go on from here knowing all the basics and with the understanding to tackle more exotic designs.
The book isn't all hardware, it also covers numbering systems, text encoding, machine code and assembler -at the lower end with a brief look at operating systems and languages at the higher end of abstraction.