mtcore::io::Reader< Impl > Struct Template Reference

A reader that reads data from some sort of stream or buffer Note: the data elements read should be trivially constructible, trivially destructible, and trivially copyable. More...

#include <reader.hpp>

Inheritance diagram for mtcore::io::Reader< Impl >:

Collaboration diagram for mtcore::io::Reader< Impl >:

Public Types
using	ReadElem = typename Impl::ReadElem
using	ErrType = typename Impl::ErrType
using	BuffSlice = Slice<std::remove_const_t<ReadElem>>
using	ResSlice = Slice<std::add_const_t<ReadElem>>

Public Member Functions
Result< ResSlice, ErrType >	read (BuffSlice buff)
	Tries to read data from a reader into a buffer It will stop once the buffer is full, or once the EOF is reached If EOF is reached, the resulting subslice may be smaller than the buff Will simply return smaller buffers if EOF is met (buffers may be empty)
Result< ResSlice, ErrType >	read_no_eof (BuffSlice buff)
	Tries to read data from a reader into a buffer It will stop once the buffer is full, or once the EOF is reached If EOF is reached, will return the error END_OF_FILE.
Result< ResSlice, ErrType >	read_all (BuffSlice buff)
	Tries to read all data from a reader into a buffer It will stop once the EOF is reached If the buffer is not big enough, will return the error SIZE_EXCEEDED.
Result< BuffSlice, ErrType >	read_all (Allocator &alloc, size_t maxSize=std::numeric_limits< size_t >::max())
	Tries to read all data from a reader into an internally allocated buffer Will try to shrink the buffer before returning (so an extra copy) It will stop once the EOF is reached If the buffer required would exceed maxSize, will return the error SIZE_EXCEEDED If there is an allocation error at some point, will return the error ALLOCATION_FAILED May return an empty buffer if there is no data to read The caller MUST deallocate the returned Slice with alloc.destroy_many()
Result< ReadElem, ErrType >	read_one ()
	Will try to read a single item from the reader If there is no item to read (aka at EOF), will return the error END_OF_FILE.
Result< void, ErrType >	skip (size_t n)
	Will skip n items (basically reads and discards them) If there are not n items left to skip (aka it hits an EOF), will return the error END_OF_FILE.

Public Attributes
Impl	underlying

Detailed Description

template<ReaderImpl Impl>
struct mtcore::io::Reader< Impl >

A reader that reads data from some sort of stream or buffer Note: the data elements read should be trivially constructible, trivially destructible, and trivially copyable.

Template Parameters

Impl	Reader Implementation (

See also

ReaderImpl concept)

Definition at line 42 of file io/reader.hpp.

Member Typedef Documentation

BuffSlice

template<ReaderImpl Impl>

using mtcore::io::Reader< Impl >::BuffSlice = Slice<std::remove_const_t<ReadElem>>

Definition at line 46 of file io/reader.hpp.

ErrType

template<ReaderImpl Impl>

using mtcore::io::Reader< Impl >::ErrType = typename Impl::ErrType

Definition at line 44 of file io/reader.hpp.

ReadElem

template<ReaderImpl Impl>

using mtcore::io::Reader< Impl >::ReadElem = typename Impl::ReadElem

Definition at line 43 of file io/reader.hpp.

ResSlice

template<ReaderImpl Impl>

using mtcore::io::Reader< Impl >::ResSlice = Slice<std::add_const_t<ReadElem>>

Definition at line 47 of file io/reader.hpp.

Member Function Documentation

read()

template<ReaderImpl Impl>

Result< ResSlice, ErrType > mtcore::io::Reader< Impl >::read ( BuffSlice buff )

inline

Tries to read data from a reader into a buffer It will stop once the buffer is full, or once the EOF is reached If EOF is reached, the resulting subslice may be smaller than the buff Will simply return smaller buffers if EOF is met (buffers may be empty)

Parameters

buff	Buffer to place read data into

Returns

Result of subslice over buff of what was read into, or the error which happened

Definition at line 59 of file io/reader.hpp.

{ return underlying.read(buff); }

Here is the caller graph for this function:

read_all() [1/2]

template<ReaderImpl Impl>

Result< BuffSlice, ErrType > mtcore::io::Reader< Impl >::read_all ( Allocator & alloc, size_t maxSize = std::numeric_limits<size_t>::max() )

inline

Tries to read all data from a reader into an internally allocated buffer Will try to shrink the buffer before returning (so an extra copy) It will stop once the EOF is reached If the buffer required would exceed maxSize, will return the error SIZE_EXCEEDED If there is an allocation error at some point, will return the error ALLOCATION_FAILED May return an empty buffer if there is no data to read The caller MUST deallocate the returned Slice with alloc.destroy_many()

Parameters

alloc	Allocator to allocate data with
maxSize	Maximum size for allocation. Default is no limit. If over 100MB, will allocate in 8,000 item chunks while reading

Returns

Result of allocated Slice on success, or the error which happened

Definition at line 113 of file io/reader.hpp.

                                                                                                           {
      ensure(maxSize > 0, "BAD MAX SIZE");
      // If we have no limit or a really high limit (> 100MB), we're going to allocate in chunks
      if (maxSize == std::numeric_limits<size_t>::max() || maxSize >= 100000000) {
        // A segmented list will be use to track the chunks
        // We'll store the actual chunks here rather than copy the individual items
        SegmentedList<BuffSlice> fullBuff;
        if (auto res = fullBuff.init(alloc, 10); res.is_error()) {
          return error(ErrType::ALLOCATION_FAILED);
        }
        // We will do a full copy before a successful return
        // So we can safely set it up to always clean up the full buff
        mtdefer {
          BuffSlice curSlice;
          auto iter = fullBuff.iter();
          while (iter.next().copy_if_present(curSlice)) {
            alloc.destroy_many(curSlice);
          }
          fullBuff.deinit(alloc);
        };
 
        size_t count = 0;
 
        bool atEnd = false;
        do {
          bool added = false;
          // We do chunks of around 8,000 items
          auto curRes = alloc.create_many<std::remove_const_t<ReadElem>>(8000);
          if (curRes.is_error()) {
            return error(ErrType::ALLOCATION_FAILED);
          }
 
          auto cur = curRes.value();
          mtdefer {
            if (!added) {
              alloc.destroy_many(cur);
            }
          };
 
          auto readRes = read(cur);
          if (readRes.is_error()) {
            return error(ErrType::ALLOCATION_FAILED);
          }
 
          auto readData = readRes.value();
          if (readData.empty()) {
            atEnd = true;
          }
          else {
            auto toPush = cur.sub(0, readData.size());
            if (auto pushRes = fullBuff.push(alloc, toPush); pushRes.is_error()) {
              return error(ErrType::ALLOCATION_FAILED);
            }
 
            added = true;
            count += readData.size();
 
            if (readData.size() < cur.size()) {
              atEnd = true;
            }
          }
        } while (!atEnd);
 
        if (count == 0) {
          return BuffSlice{nullptr, 0};
        }
 
        // We have everything chunked in memory, but our contract is to return a contiguous set of bytes
        // To do this, we need to do one final copy into contiguous memory
        auto finalRes = alloc.create_many<std::remove_const_t<ReadElem>>(count);
        if (finalRes.is_error()) {
          return error(ErrType::ALLOCATION_FAILED);
        }
 
        auto finalSlice = finalRes.value();
        // We'll use a Slice writer to make things a little simpler
        auto finalWriter = slice_writer(finalSlice);
        // Since we stored the chunks, we can just iterate each chunk and write the entire chunk at once
        auto iter = fullBuff.iter();
        BuffSlice curBuff;
        while (iter.next().copy_if_present(curBuff)) {
          if (auto r = finalWriter.write_all(curBuff); r.is_error()) {
            alloc.destroy_many(finalSlice);
            return error(ErrType::ALLOCATION_FAILED);
          }
        }
 
        // Return our Result
        return finalSlice;
      }
      else {
        // If we have a smal enough max size, just read it all into one chunk of memory
        auto outRes = alloc.create_many<std::remove_const_t<ReadElem>>(maxSize);
        if (outRes.is_error()) {
          return error(ErrType::ALLOCATION_FAILED);
        }
        auto out = outRes.value();
        auto res = read_all(out);
        if (res.is_error()) {
          alloc.destroy_many(out);
          return res.error();
        }
        auto finalRes = res.value();
 
        // Check to see if we can save memory if we shrink things down
        // This is more CPU time, but less memory pressure
        if (finalRes.size() + alignof(ReadElem) < out.size()) {
          if (finalRes.size() == 0) {
            alloc.destroy_many(out);
            return BuffSlice{nullptr, 0};
          }
          // Try to create a sub optimal buffer
          auto smallerRes = alloc.create_many<std::remove_const_t<ReadElem>>(finalRes.size());
          if (smallerRes.is_error()) {
            // we have it already in memory, it's just suboptimal. Return it anyway
            return out.sub(0, finalRes.size());
          }
          auto smaller = smallerRes.value();
          auto w       = slice_writer(smaller);
          ensure(w.write_all(finalRes).is_success(), "SOMETHING IS WRONG WITH SLICE WRITER!");
          alloc.destroy_many(out);
          return success(smaller);
        }
        return success(out.sub(0, finalRes.size()));
      }
    }

Here is the call graph for this function:

read_all() [2/2]

template<ReaderImpl Impl>

Result< ResSlice, ErrType > mtcore::io::Reader< Impl >::read_all ( BuffSlice buff )

inline

Tries to read all data from a reader into a buffer It will stop once the EOF is reached If the buffer is not big enough, will return the error SIZE_EXCEEDED.

Parameters

buff	Buffer to place read data into

Returns

Result of subslice over buff of what was read into, or the error which happened

Definition at line 83 of file io/reader.hpp.

                                                       {
      auto res = read(buff);
      if (res.is_error()) {
        return res.error();
      }
      auto resSlice = res.value();
      if (resSlice.size() < buff.size()) {
        return resSlice;
      }
 
      // If there's another element not read, then we didn't read everything
      if (auto ro = read_one(); ro.is_success() || ro.error().code != ErrType::END_OF_FILE) {
        return error(ErrType::SIZE_EXCEEDED);
      }
      return resSlice;
    }

Here is the call graph for this function:

Here is the caller graph for this function:

read_no_eof()

template<ReaderImpl Impl>

Result< ResSlice, ErrType > mtcore::io::Reader< Impl >::read_no_eof ( BuffSlice buff )

inline

Tries to read data from a reader into a buffer It will stop once the buffer is full, or once the EOF is reached If EOF is reached, will return the error END_OF_FILE.

Parameters

buff	Buffer to place read data into

Returns

Result of subslice over buff of what was read into, or the error which happened

Definition at line 68 of file io/reader.hpp.

                                                          {
      auto res = read(buff);
      if (res.is_success() && res.value().size() < buff.size()) {
        return error(ErrType::END_OF_FILE);
      }
      return res;
    }

Here is the call graph for this function:

read_one()

template<ReaderImpl Impl>

Result< ReadElem, ErrType > mtcore::io::Reader< Impl >::read_one ( )

inline

Will try to read a single item from the reader If there is no item to read (aka at EOF), will return the error END_OF_FILE.

Returns

Result of one item read, or error

Definition at line 245 of file io/reader.hpp.

                                         {
      std::remove_const_t<ReadElem> elem;
      auto s       = Slice<std::remove_const_t<ReadElem>>{&elem, 1};
      auto readRes = read(s);
      if (readRes.is_error()) {
        return readRes.error();
      }
      auto res = readRes.value();
      if (res.empty()) {
        return error(ErrType::END_OF_FILE);
      }
      return elem;
    }

Here is the call graph for this function:

Here is the caller graph for this function:

skip()

template<ReaderImpl Impl>

Result< void, ErrType > mtcore::io::Reader< Impl >::skip ( size_t n )

inline

Will skip n items (basically reads and discards them) If there are not n items left to skip (aka it hits an EOF), will return the error END_OF_FILE.

Parameters

n	Number of items to skip

Returns

Error if there was one

Definition at line 265 of file io/reader.hpp.

                                         {
      for (size_t i = 0; i < n; ++i) {
        auto r = read_one();
        if (r.is_error()) {
          return r.error();
        }
      }
      return success();
    }

Here is the call graph for this function:

Member Data Documentation

underlying

template<ReaderImpl Impl>

Impl mtcore::io::Reader< Impl >::underlying

Definition at line 49 of file io/reader.hpp.

---

The documentation for this struct was generated from the following file:

• io/reader.hpp